:W&Bá¾~ñtM 6ljjlx35ª; 9 /0.28.1+bc2cf24b1bd9fa3a1b6cbeb920087b832afc526f0.70.0Hä/„‚" yrj8wd3xxo6r 6ljjlx35Šà 6ljjlx35escheuller-uc-san-diego spatop-sweep"Ÿ _wandb‚{}  batch_size‚2048  dropout‚0.23316927734185028  hidden_dim‚256 ( learning_rate‚0.00048247339625462246  num_embedding_layers‚6Bdpfx_vanilla_6ljjlx35bnqupkqclj%dp-spatop-sweepagent-vanilla-v2-h9cgsŠ îìêÒ˜ñ¨øš3 (H¨¸hp"3.12.10*0.28.1j linux-x86_64yi‘ó; 6ljjlx35*  _wandb‚ {"runtime":0}  _runtime‚0»ÑÏ*bÒÜ -Linux-6.8.0-134-generic-x86_64-with-glibc2.36CPython 3.12.10 îìêÒ°þù±*-of*>/data/spatop/options_files/sweep_fixed_dp/config_6ljjlx35.json*-l* /scratch/logs*-n*dpfx_vanilla_6ljjlx352-m spanet.trainRdanielprimosch@gmail.comZ /scratch/logsb%dp-spatop-sweepagent-vanilla-v2-h9cgsr/opt/conda/bin/pythonº swlo5ukt7dhra6qz3mhkhhc7klvznspiO¤=ÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€à†å“ª€àçÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-0c583aac-6f68-6140-7078-f37971b4b9bbÊ13.2º swlo5ukt7dhra6qz3mhkhhc7klvznspiµª½Ç2  requirements.txtKœÈ¿W 6ljjlx35ZF (H¨¸ (H¨¸ (hpð"3.12.10*0.28.1j linux-x86_64U³4Y 6ljjlx35ZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.28.1j linux-x86_64&‚%*  6ljjlx35b trainer/global_step2m•©â\  6ljjlx35ZK (H¨¸ (H¨¸(8hpð"3.12.10*0.28.1j linux-x86_64aËÛÁ/  6ljjlx35b*"trainer/global_step2´TuÐK  6ljjlx35j: ïìêÒè¬ýè*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] ôÄ•“  6ljjlx35j  ïìêÒ Ôžõî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 3.4 M │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 5.3 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 75.8 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ =°uu\ 6ljjlx35jÊ ïìêÒÀü˜ö·Trainable params: 84.5 M Non-trainable params: 50 Total params: 84.5 M Total estimated model params size (MB): 337.922 Modules in train mode: 2299 Modules in eval mode: 0 Total FLOPs: 0 Ï8‡q 6ljjlx35j` ïìêÒ€½ÂöPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). hÕ{ 6ljjlx35jj ïìêÒÈÊöZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). ëb˜q 6ljjlx35j` ïìêÒПÑöPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). M]¸ft 6ljjlx35jc ïìêÒ€û×öSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). VžÆvn 6ljjlx35j] ïìêÒ¨¯ÞöMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). ŽÑ,=ð 6ljjlx35*Þ  transformer_dim‚256  transformer_dim_scale‚2.0  initial_embedding_dim‚256  position_embedding_dim‚256  num_encoder_layers‚4 ! num_branch_embedding_layers‚3  num_branch_encoder_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1  num_attention_heads‚8 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true # use_pairwise_interactions‚false  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚""  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 K event_info_file‚7"/data/spatop/event_files/v11/tt_hadronic_v7_full.yaml" F training_file‚4"/data/spatop/tpm70_wpm30_FB350_fixed/all_merged.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_limit‚1.0  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 epochs‚50 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"² ¹ë 6ljjlx35jÙ ïìêÒÀœ¬úÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ŠHjN 6ljjlx35j¼ ïìêÒØçºú«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. x„gë 6ljjlx35jÙ óìêÒ þ³–È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. {®ÝåP 6ljjlx35j¾ óìêÒðÈÊ–­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ðÏi’‚(:ˆ þìêÒ݇ÜÇ gpu.0.gpu‚74.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚57.74421691894531* gpu.0.memoryAllocatedBytes‚ 49601904640 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uþ’½Ä‚(:º þìêÒÏÞ©Î disk.sda3.out‚ 0.60546875 disk.rbd0.in‚0) disk./.usagePercent‚38.98787021966348 disk.nvme0c0n1.in‚ 0.83203125$ disk.nvme0c0n1.out‚ 8.79248046875 disk.rbd2.in‚0 disk.sda2.out‚0) proc.memory.percent‚16.85352325439453 disk.dm-0.in‚ 0.83203125 disk.rbd2.out‚0 disk.rbd1.out‚0 proc.memory.rssMB‚ 2761.28125 cpu‚25.924694538809455 network.recv‚41315 disk.nvme0n1.in‚ 0.83203125" disk.nvme0n1.out‚ 8.79248046875 disk.sda3.in‚0 disk.sda.out‚ 0.60546875$ memory_percent‚86.92588806152344 proc.cpu.threads‚17 network.sent‚33109 disk.rbd0.out‚0 disk.sda.in‚0 disk.drbd2165.in‚0% proc.memory.availableMB‚ 2142.0625 disk.sda1.in‚0 disk.dm-1.out‚ 0.60546875$ disk./.usageGB‚580.9474411010742 disk.dm-0.out‚ 8.79248046875 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0w%×’‚(:ˆ íêÒÿë¢Ç gpu.0.gpu‚99.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.74665832519531* gpu.0.memoryAllocatedBytes‚ 49604001792 gpu.0.temp‚40.0 gpu.0.powerWatts‚299.456' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.8912 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pPYÓÕ‚(:Ë íêÒ…†•É disk.dm-0.out‚15.01806640625 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚580.9472808837891 disk.rbd1.in‚0 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2141.55859375 disk.nvme0n1.in‚ 0.83203125 disk.sda1.in‚0 disk.sda3.out‚ 0.94140625 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.rbd2.out‚0 disk.sda2.out‚0 disk.rbd1.out‚0.0625 network.recv‚45408* disk./.usagePercent‚38.987859467347036 disk.dm-1.out‚ 0.94140625 cpu‚25.769085620524006 disk.sda.out‚ 0.94140625$ memory_percent‚86.92896366119385 disk.dm-0.in‚ 0.83203125 disk.nvme0c0n1.in‚ 0.83203125# proc.memory.rssMB‚ 2761.31640625* proc.memory.percent‚16.853737831115723# disk.nvme0n1.out‚15.01806640625 disk.dm-1.in‚0% disk.nvme0c0n1.out‚15.01806640625 network.sent‚41729´}óž‚(:” œíêÒÇÕÄÇ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.749099731445305* gpu.0.memoryAllocatedBytes‚ 49606098944 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.794' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.358799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø¡qŽÊ‚(:À œíêÒ¹ö£Ê disk.sda3.out‚1.171875 disk.sda2.out‚0 disk.sda.out‚1.171875 proc.memory.rssMB‚ 2761.34375* proc.memory.percent‚16.853904724121094 disk.nvme0n1.in‚ 51.87890625 disk.rbd2.in‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 2141.30859375 network.recv‚49246 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.out‚ 23.1845703125 disk.rbd0.in‚0 disk.dm-1.out‚1.171875 disk.sda.in‚0! disk.nvme0c0n1.in‚ 51.87890625# memory_percent‚86.9304895401001 network.sent‚47108) disk./.usagePercent‚38.98786151540731 cpu‚25.689367136907006 disk.dm-0.out‚ 23.1845703125$ disk./.usageGB‚580.9473114013672 disk.sda1.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.rbd1.out‚0.0625 proc.cpu.threads‚17 disk.rbd2.out‚0" disk.nvme0n1.out‚ 23.1845703125 disk.rbd0.out‚0 disk.dm-0.in‚ 51.87890625VçV’‚(:ˆ «íêÒñÔ‹É gpu.0.gpu‚99.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚57.75154113769532* gpu.0.memoryAllocatedBytes‚ 49608196096 gpu.0.temp‚37.0 gpu.0.powerWatts‚111.241' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2482 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™b9©Ð ‚(:Æ «íêÒ”–¯É disk.sda.in‚0) disk./.usagePercent‚38.98785716327923$ disk./.usageGB‚580.9472465515137 disk.rbd0.out‚0 disk.rbd1.out‚0.140625 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 144.76953125 disk.dm-0.out‚ 27.9365234375" disk.nvme0n1.out‚ 27.9365234375 disk.rbd1.in‚0 disk.sda.out‚ 1.53515625% proc.memory.availableMB‚ 2141.3125 disk.rbd2.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 144.76953125* proc.memory.percent‚16.854095458984375 disk.sda1.in‚0 network.sent‚52595 disk.sda3.in‚0 disk.dm-1.out‚ 1.53515625$ disk.nvme0c0n1.out‚ 27.9365234375 proc.cpu.threads‚17 disk.sda3.out‚ 1.53515625 disk.rbd2.in‚0 proc.memory.rssMB‚2761.375 cpu‚25.639001456223184 network.recv‚53252 disk.nvme0n1.in‚ 144.76953125 disk.rbd0.in‚0 disk.sda2.in‚0$ memory_percent‚86.93046569824219 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0ˆÓ*è!‚(:“ ºíêÒ˜ÄÐÇ gpu.0.gpu‚98.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.75154113769532* gpu.0.memoryAllocatedBytes‚ 49608196096 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«DYíÉ"‚(:¿ ºíêÒ¡™Ê disk.sda1.in‚0 disk.sda3.out‚1.9375 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda.out‚1.9375 disk.dm-0.out‚36.32080078125 network.sent‚58336) disk./.usagePercent‚38.98787175570868 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0! proc.memory.rssMB‚ 2761.390625 disk.rbd2.out‚0% disk.nvme0c0n1.out‚36.32080078125 disk.drbd2165.out‚0 network.recv‚56946# disk.nvme0n1.out‚36.32080078125 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚0.140625 disk.dm-0.in‚ 145.6015625$ memory_percent‚86.93041801452637 disk.sda2.out‚0 disk.dm-1.out‚1.9375 cpu‚25.60527484385713 disk.nvme0n1.in‚ 145.6015625 disk.sda.in‚0! disk.nvme0c0n1.in‚ 145.6015625 proc.cpu.threads‚17$ disk./.usageGB‚580.9474639892578( proc.memory.availableMB‚ 2141.3203125* proc.memory.percent‚16.854190826416016f"õå3#b/ lr-AdamW/pg1"trainer/global_step2:JPi¢3$b/ lr-AdamW/pg2"trainer/global_step2:JP¹Ñ4 -%) _wandbruntime‚86  _runtime‚86Ú½ [À&» _runtime‚ 86.691428514 _step‚0 & lr-AdamW/pg1‚5.910299104119125e-05 & lr-AdamW/pg2‚5.910299104119125e-05 trainer/global_step‚49 % _timestamp‚1.7843299261062543e+09Ë:»Â'½ trainer/global_step‚49 % _timestamp‚1.7843299261062543e+09 _runtime‚ 86.691428514 _step‚0 & lr-AdamW/pg1‚5.910299104119125e-05 & lr-AdamW/pg2‚5.910299104119125e-056d\Ys(bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPnmk)bg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP5Tcâg*bc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JPcDÔjj+bf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JPßØc£f,bb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JPöC èi-be BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JP÷_‡o.bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPö,uöW/bS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JPfÅãúf0bb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP:žx¹l1bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPGãåRj2bf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JPi»T5q3bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP²×Öl4bh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPà'DPx5bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPQ^Ûui6be BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JP0¦Ap7bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JP¼â¿m8bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP­JÈg9bc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP_„Ê¡i:be BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JP`Cìj;bf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JP‹Ëd½kbh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP:»9þq?bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JP·0gv@br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPOYúÑiAbe BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JP¯œùdrBbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP ¿ägCbc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JP… í}nDbj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JPàýh7iEbe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JPužyhFbd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JP†KÑAqGbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPF ›gHbc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP9%?gIbc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JP ”hÒpJbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPU1»xKbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPŒˆ‡¡jLbf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JPd­d£nMbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP«r¡êkNbg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JP_U nObj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP‘`÷iPbe BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JP”bP¨pQbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP§·2¢fRbb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JPƈbOsSbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP­UUó`Tb\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JP”ó+‹aUb] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JP.ç_Vb[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JPM©€^WbZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPòeN…lXbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPÆš(“nYbj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPF ܸtZbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP•rŽ¤^[bZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JPB^…na\b] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPz±W"h]bd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JPÓïY]d^b` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JP°«›h_bd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JPãb+e`ba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPG)SÀsabo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPNJQobbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP0B +ncbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPL¹gdbc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JP¢•‚mebi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JP9„humfbi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JPÀÓ&fgbb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JP›æghbc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JP—#÷ófibb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JP`øDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPæp¹’iobe BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JP—ÄgKjpbf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JP¢¤äxfqbb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JP„AJ/orbk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPrÞäjsbf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JP¦ô>Ìktbg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JP¯V+ publ IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPf“ ·lvbh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPüñiëhwbd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JP•CO¤vxbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPëÁÿÄ_yb[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JP á¡–czb_ DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP‘":Þn|bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JP áÙb}b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JP²kÿl~bh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JPR,Ùnbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JP¥ÌÛo€bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPfüNpbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPAдºc‚b^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JPÔݬqƒbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP ‹‘Un„bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPïé›r…bm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPÊÊgd†b_ DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPÏ+o“bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPâÓ<]t”bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPΰübh•bc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JPh°@u–bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPV"p—bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPV ¤ør˜bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPõ¨i×s™bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPljošbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP¹ø`›b[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JP•€>Jiœbd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JP í’fba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPî÷êÿtžbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPQ×ÀAsŸbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JPšºy9j be BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JP¢e‹Ko¡bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP[Š|ol¢bg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JP[¹Ü.£) _wandbruntime‚86  _runtime‚86•Ù¸e¤ŸO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544943616 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14032387673088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 233325056 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 45531893768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚269494 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4126860197376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469583 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚817 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚414665 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14032387673088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚422323 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚502 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402807296 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 46956409344 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14033759056384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34716160 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1992 GDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11744 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1992 % _timestamp‚1.7843299261076927e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚132403 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚13986776473600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11801 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4127263004672 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679936000 trainer/global_step‚49 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 46956409344 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚846616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4065335644160 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13980577312768 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 _step‚1 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚137351 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1992 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14026109206536 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚13980602740736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1713373184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚817 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49064968192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚414671 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚422323 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11744 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 45611199488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49064968192 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 45611199488 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1280991744 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 61559269376 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚817 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1713373184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544943616 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14033759056384 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚431945 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚132407 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14027479233298 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49064968192 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚414671 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4065703735296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚137091 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13986802647040 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚414665 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13986802647040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚836988 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11801 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13986776473600 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚431945 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 61524553216@ à©'¥£O D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1992 % _timestamp‚1.7843299261076927e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚132403 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚13986776473600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11801 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4127263004672 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679936000 trainer/global_step‚49 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 46956409344 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚846616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4065335644160 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13980577312768 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 _step‚1 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚137351 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1992 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14026109206536 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚13980602740736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1713373184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚817 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49064968192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚414671 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚422323 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11744 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 45611199488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49064968192 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 45611199488 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1280991744 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 61559269376 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚817 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1713373184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544943616 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14033759056384 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚431945 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚132407 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14027479233298 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49064968192 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚414671 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4065703735296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚137091 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13986802647040 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚414665 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13986802647040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚836988 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11801 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13986776473600 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚431945 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 61524553216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544943616 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14032387673088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 233325056 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 45531893768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚269494 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4126860197376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469583 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚817 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚414665 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14032387673088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚422323 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚502 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402807296 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 46956409344 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14033759056384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34716160 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1992 GDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11744 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856§Òg¦bb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JP(È…1e§b` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPžË¤g¨bb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JPq5Í]©bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPªÈÝvhªbc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JPP¿Î l«bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPVÿh¬bc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JPk©üèe­b` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPg$áÅi®bd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JP пl¯bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPÃŽ¨ƒd°b_ DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP2i¸bd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JPnè)m¹bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPnä)oºbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JP›ð5©^»bY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JPÚ¹Qi¼bd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JPì”ô^½bY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JP‡ºci¾bd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JPcÅÿp¿bk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPÈ—UeÀb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JPJp̪rÁbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPKÁ8mÂbh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP*8ú mÃbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPr¨õžuÄbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP 艉\ÅbW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JPr#Ú`Æb[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPù=•×gÇbb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JPÇMHÚbÈb] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JPéá}QkÉbf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JP²êjÊbe BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JP¦†ÏveËb` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JPŠíFDfÌba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPõZcÑqÍbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPDçi‘lÎbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPgžDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPd§t¥lábg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JP¶I,Àmâbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JP ’Æ+sãbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPÇšøôräbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP:ž·Ékåbf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPg|Ã2wæbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP[+Šsçbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPZ¦Âkèbf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPÊÜäJhébc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JPЋ :dêb_ Ûrýbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP³¤²‚fþba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP &Xcÿb^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPõçÒj€be BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPnMÞ7ab\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JPøó“·j‚be BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JPŒ‰Ifƒba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP?SXÎl„bg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JPFÿÕ*n…bi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JPÑOgm†bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPgÂiP`‡b[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPÖe@Èhˆbc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JP[žjÚp‰bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPÑEakŠbf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JPéÇÁC]‹bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JP”HÏoŒbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JPÕÑxmdb_ DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPÿäZfba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPâL5Òm‘bh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JP0s©¤o’bj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JPH7ÉZm“bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JP½2e”b` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JPOéåço•bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPëÊZh–bc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JPÔâÅp—bk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JP9w?ìb˜b] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JPØ>j™be BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JP „¾?fšba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP»æºk›bf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPu­qÓgœbb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPóÂ9Lrbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JP­8ýÓažb\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JPß°YnŸbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JP(LøyV bQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JPª˜l.¡) _wandbruntime‚88  _runtime‚88µ½£¹¢©M WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 47881452032 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14312686926336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚274810 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚139867 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4142824148992 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1280991744 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 62759310336 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 62724601344 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14306284477832 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1992 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14258459090944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚422871 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14264805474304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4205951550464 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚863314 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚817 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843299275210006e+09 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 46536235008 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14311315535872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14258484518912 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14264779300864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚475 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49064968192 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49064968192 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚430801 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚863314 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1713373184 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚853672 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚422871 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1992 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚502 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 237518336 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚422877 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚440437 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚853672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚275070 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚817 trainer/global_step‚49 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1713373184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚422877 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14312686926336 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 46536235008 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4205548750336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6-óƒ!³&£­M RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1280991744 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 62759310336 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 62724601344 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14306284477832 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1992 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14258459090944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚422871 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14264805474304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4205951550464 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚863314 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚817 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843299275210006e+09 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 46536235008 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14311315535872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14258484518912 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14264779300864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚475 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49064968192 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49064968192 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚430801 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚863314 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1713373184 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚853672 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚422871 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1992 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚502 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 237518336 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚422877 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚440437 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚853672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚275070 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚817 trainer/global_step‚49 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1713373184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚422877 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14312686926336 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 46536235008 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4205548750336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 47881452032 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14312686926336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚274810 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚139867 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4142824148992 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640ØB¤b= loss/SRqqt2/detection_loss"trainer/global_step2:JP¡|̪C¥b> loss/SRqqt1/assignment_loss"trainer/global_step2:JPjðuU@¦b; loss/FRt1/detection_loss"trainer/global_step2:JPM›2@§b; loss/FBt1/detection_loss"trainer/global_step2:JPŸõ' -¨b( epoch"trainer/global_step2:JP]_˜VA©b< loss/FBt1/assignment_loss"trainer/global_step2:JP™ ª]Bªb= loss/SRqqt1/detection_loss"trainer/global_step2:JP++@«b; loss/FBt2/detection_loss"trainer/global_step2:JP‚ê7¬b2 loss/total_loss"trainer/global_step2:JPÈ20ßC­b> loss/SRqqt2/assignment_loss"trainer/global_step2:JP¡L‹gA®b< loss/FRt1/assignment_loss"trainer/global_step2:JPfMœ¿A¯b< loss/FRt2/assignment_loss"trainer/global_step2:JP©Óo²A°b< loss/FBt2/assignment_loss"trainer/global_step2:JPL+ä|@±b; loss/FRt2/detection_loss"trainer/global_step2:JPqEk.²) _wandbruntime‚88  _runtime‚88)ÚÑô³î _step‚3 /loss/FRt1/assignment_loss‚4.536730899443489 .loss/FRt2/assignment_loss‚5.34806850591876 0loss/FBt2/assignment_loss‚0.8363075602984598 .loss/FRt2/detection_loss‚1.438386056370699 0loss/SRqqt2/detection_loss‚1.587839287558376 trainer/global_step‚49 $ _timestamp‚1.784329927592498e+09 1loss/SRqqt1/assignment_loss‚4.231709364755654 /loss/FRt1/detection_loss‚0.3964515258263854 .loss/FBt1/detection_loss‚50.69110348820686 epoch‚0 0loss/FBt1/assignment_loss‚0.8489944083350045 1loss/SRqqt1/detection_loss‚3.9682256627689907 .loss/FBt2/detection_loss‚7.693110193835298 $loss/total_loss‚85.7351323287047 1loss/SRqqt2/assignment_loss‚4.158205375386708 _runtime‚ 88.177637611ÚQ“Óø´ò 1loss/SRqqt2/assignment_loss‚4.158205375386708 _runtime‚ 88.177637611 _step‚3 /loss/FRt1/assignment_loss‚4.536730899443489 .loss/FRt2/assignment_loss‚5.34806850591876 0loss/FBt2/assignment_loss‚0.8363075602984598 .loss/FRt2/detection_loss‚1.438386056370699 0loss/SRqqt2/detection_loss‚1.587839287558376 trainer/global_step‚49 $ _timestamp‚1.784329927592498e+09 1loss/SRqqt1/assignment_loss‚4.231709364755654 /loss/FRt1/detection_loss‚0.3964515258263854 .loss/FBt1/detection_loss‚50.69110348820686 epoch‚0 0loss/FBt1/assignment_loss‚0.8489944083350045 1loss/SRqqt1/detection_loss‚3.9682256627689907 .loss/FBt2/detection_loss‚7.693110193835298 $loss/total_loss‚85.7351323287047µ2䑵‚(:† ÉíêÒÂòÎÇ gpu.0.gpu‚99.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.75398254394531* gpu.0.memoryAllocatedBytes‚ 49610293248 gpu.0.temp‚42.0 gpu.0.powerWatts‚269.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨?ñQÓ¶‚(:È ÉíêÒøã¨Ê) disk./.usagePercent‚38.98787277973882 disk.rbd0.out‚0 disk.rbd1.out‚0.265625 disk.sda.out‚ 2.23046875 disk.sda3.in‚0# proc.memory.rssMB‚ 2761.44140625 network.recv‚60970 disk.dm-0.in‚ 146.55859375 disk.sda3.out‚ 2.23046875 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 146.55859375 disk.sda1.in‚0 cpu‚25.582829449502636! disk.nvme0n1.out‚ 46.248046875 disk.rbd2.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 2.23046875 disk.sda.in‚0$ memory_percent‚86.94190979003906% proc.memory.availableMB‚ 2139.4375* proc.memory.percent‚16.854500770568848 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.sent‚63708 disk.rbd2.in‚0$ disk./.usageGB‚580.9474792480469 disk.dm-0.out‚ 46.248046875 disk.sda1.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 146.55859375# disk.nvme0c0n1.out‚ 46.248046875Kk‘·‚(:† ØíêÒ¯ÖÄÇ gpu.0.gpu‚99.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.75398254394531* gpu.0.memoryAllocatedBytes‚ 49610293248 gpu.0.temp‚40.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬…R½Í¸‚(: ØíêÒûÛÀÉ disk.sda.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 146.55859375 disk.dm-0.out‚ 49.66796875 disk.sda2.in‚0) proc.memory.availableMB‚ 2139.19921875 network.sent‚117395 disk.nvme0n1.out‚ 49.66796875 disk.sda1.in‚0 disk.sda2.out‚0) disk./.usagePercent‚38.98786561152786 disk.rbd1.out‚0.265625 disk.sda3.in‚0 disk.rbd2.out‚0$ memory_percent‚86.94336414337158 disk.sda.out‚2.421875" disk.nvme0c0n1.in‚ 146.55859375 disk.sda3.out‚2.421875 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0) proc.memory.percent‚16.85464382171631 cpu‚25.56018136265364 proc.cpu.threads‚17 network.recv‚67866$ disk./.usageGB‚580.9473724365234 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2761.46484375" disk.nvme0c0n1.out‚ 49.66796875 disk.nvme0n1.in‚ 146.55859375 disk.dm-1.out‚2.421875;ia¹‚(:’ çíêÒÊêåÇ gpu.0.gpu‚99.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.75642395019531* gpu.0.memoryAllocatedBytes‚ 49612390400 gpu.0.temp‚41.0 gpu.0.powerWatts‚295.687' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.13740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷ë¸Êº‚(:¿ çíêÒ“™²Ê network.sent‚122878 network.recv‚72021$ disk./.usageGB‚580.9473838806152 disk.rbd1.in‚0 disk.sda1.in‚0 disk.dm-0.out‚ 60.6669921875 disk.sda3.in‚0 disk.rbd0.out‚0) proc.memory.percent‚16.85466766357422 disk.sda2.out‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.out‚ 60.6669921875 disk.nvme0n1.in‚ 202.734375# memory_percent‚86.9452953338623 disk.rbd2.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 202.734375" disk.nvme0n1.out‚ 60.6669921875 proc.memory.rssMB‚ 2761.46875) disk./.usagePercent‚38.98786637955046 disk.sda2.in‚0 disk.dm-1.out‚ 2.6640625 disk.rbd0.in‚0 disk.sda.out‚ 2.6640625 disk.sda1.out‚0 cpu‚25.54331164081426 disk.drbd2165.in‚0 disk.dm-0.in‚ 202.734375 disk.sda3.out‚ 2.6640625( proc.memory.availableMB‚ 2138.8828125 disk.rbd2.out‚0 disk.rbd1.out‚0.34375 disk.sda.in‚0 proc.cpu.threads‚17ÓkF¾ž»‚(:“ öíêÒ‡‰È gpu.0.gpu‚98.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.75642395019531* gpu.0.memoryAllocatedBytes‚ 49612390400 gpu.0.temp‚40.0 gpu.0.powerWatts‚267.174' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.434799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C-DÕ¼‚(:Ê öíêÒ­ÓÅÊ disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚0.34375) disk./.usagePercent‚38.98782439431485 disk.nvme0n1.in‚ 203.69140625# disk.nvme0n1.out‚67.79736328125 disk.sda3.out‚2.953125% disk.nvme0c0n1.out‚67.79736328125) proc.memory.availableMB‚ 2138.62109375" proc.memory.rssMB‚ 2761.4765625 cpu‚25.531399708341365 network.sent‚128399" disk.nvme0c0n1.in‚ 203.69140625 network.recv‚75777 disk.rbd2.in‚0$ memory_percent‚86.94689273834229 proc.cpu.threads‚17 disk.dm-0.out‚67.79736328125 disk.sda2.in‚0$ disk./.usageGB‚580.9467582702637 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 203.69140625 disk.dm-1.out‚2.953125 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda.out‚2.953125 disk.drbd2165.out‚0) proc.memory.percent‚16.85471534729004 disk.sda.in‚0 disk.sda1.in‚0Ëô_ᓽ‚(:ˆ …îêÒËÛ¹Ç gpu.0.gpu‚99.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.75642395019531* gpu.0.memoryAllocatedBytes‚ 49612390400 gpu.0.temp‚43.0 gpu.0.powerWatts‚128.353' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.6706 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C«‘˜Õ¾‚(:Ê …îêÒ¼ ÕÉ cpu‚25.518989888986802 disk.dm-1.out‚ 3.2578125 disk.sda.in‚0" disk.nvme0c0n1.in‚ 203.69140625 disk.dm-0.in‚ 203.69140625" disk.nvme0n1.out‚ 72.1904296875 disk.rbd0.out‚0) proc.memory.percent‚16.85473918914795 network.sent‚133882 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.dm-0.out‚ 72.1904296875 disk.sda1.in‚0 disk.sda3.out‚ 3.2578125 disk.rbd1.out‚0.40625 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 203.69140625 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda.out‚ 3.2578125 disk.rbd2.out‚0$ memory_percent‚86.94713115692139) proc.memory.availableMB‚ 2138.58203125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda3.in‚0 network.recv‚79673$ disk.nvme0c0n1.out‚ 72.1904296875 disk.sda2.in‚0# proc.memory.rssMB‚ 2761.48046875) disk./.usagePercent‚38.98784692297786# disk./.usageGB‚580.947093963623²·&i¿‚(:’ ”îêÒŽø³È gpu.0.gpu‚99.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.75642395019531* gpu.0.memoryAllocatedBytes‚ 49612390400 gpu.0.temp‚44.0 gpu.0.powerWatts‚291.095' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.21900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t %wÇÀ‚(:¼ ”îêÒùžíÉ disk.sda1.out‚0# proc.memory.rssMB‚ 2761.48828125) disk./.usagePercent‚38.98784333887238 cpu‚25.508239690630283 proc.cpu.threads‚17 network.sent‚139530 disk.dm-1.out‚3.40625! disk.nvme0c0n1.in‚ 204.5234375 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.dm-0.out‚ 79.919921875 disk.sda3.out‚3.40625# memory_percent‚86.9471549987793' proc.memory.availableMB‚ 2138.578125 network.recv‚83301 disk.rbd1.in‚0 disk.rbd1.out‚0.40625 disk.sda.in‚0 disk.sda.out‚3.40625# disk.nvme0c0n1.out‚ 79.919921875 disk.dm-1.in‚0 disk.drbd2165.in‚0! disk.nvme0n1.out‚ 79.919921875$ disk./.usageGB‚580.9470405578613 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 204.5234375 disk.rbd2.out‚0 disk.sda1.in‚0) proc.memory.percent‚16.85478687286377 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 204.52343756°cÿ0Á+ _wandbruntime‚170  _runtime‚170tÞ]ý $ _timestamp‚1.784330009688319e+09 _runtime‚ 170.273383516 _step‚4 ' lr-AdamW/pg1‚0.00011941216557301906 ' lr-AdamW/pg2‚0.00011941216557301906 trainer/global_step‚99xGvÇÃÁ _runtime‚ 170.273383516 _step‚4 ' lr-AdamW/pg1‚0.00011941216557301906 ' lr-AdamW/pg2‚0.00011941216557301906 trainer/global_step‚99 $ _timestamp‚1.784330009688319e+09†è0Ä+ _wandbruntime‚170  _runtime‚170EÌ}ŦO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1715470336 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚848075 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27886944010240 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚857697 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27980216350208 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 93272339968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1715470336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚824965 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27886917836800 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚27978844966912 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 93272339968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27874691649536 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404904448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚857697 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚259407 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7939760885760 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 120898161152 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27967805255206 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27978844966912 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚824965 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚848075 TGDeviceStatsMonitor.on_trainÉþI/_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1715470336 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚519 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚818 % _timestamp‚1.7843300096895835e+09 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚824971 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1993 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27874666221568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚536742 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚546 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 120861347840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 91769006876 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1993 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 91927130112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚818 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚818 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27886917836800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚824971 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚277625 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚277339 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1715470336 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚519 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚818 % _timestamp‚1.7843300096895835e+09 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚824971 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1993 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27874666221568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚536742 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚546 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 120861347840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 91769006876 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1993 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 91927130112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚818 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚818 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27886917836800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚824971 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚277625 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚277339 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27978844966912 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚824965 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚848075ç=áó0Ç+ _wandbruntime‚171  _runtime‚171i €µ&ȯM SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 92852165632 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚542357 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 _runtime‚ 171.683978691 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28259144220160 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28246610499740 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚866189 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1314545152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8017617481728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8017249390592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36806144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚833171 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 $ _timestamp‚1.784330011098558e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1993 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚818 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 122037963776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1715470336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚519 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚28245240472866 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1715470336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28164946837504 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1689724 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404897280 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1993 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28152573427712 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 94197382656 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚856553 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1993 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1715470336 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 94197382656 trainer/global_step‚99 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28164920664064 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 122074769920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚818 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚818 _step‚6 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1689724 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚542076 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚261943 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1699366 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 270481920 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚833171 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 $ _timestamp‚1.784330011098558e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1993 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚818 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 122037963776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1715470336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚519 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚28245240472866 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1715470336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28164946837504 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1689724 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404897280 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1993 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28152573427712 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 94197382656 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚856553 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1993 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1715470336 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 94197382656 trainer/global_step‚99 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28164920664064 `ODeviceStatsMonitor.Õ5ø; on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 122074769920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚818 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚818 _step‚6 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1689724 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚542076 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚261943 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1699366 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 270481920 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41925302260736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚41908131277114 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1717567488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49069162496 _step‚9 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49069162496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1994 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7843300930144298e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1994 trainer/global_step‚149 _runtime‚ 253.602093883 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚416390 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1235265 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚819 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1283449 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41768755130368 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1235265 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚803086 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1994 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49069162496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1717567488 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1235271 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚386407 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1994 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 138006119984 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41926673644032 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚11993735094784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41906761250352 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1283449 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38910464 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11814186127360 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 179548967424 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚41787059200000 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚558 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 179587877888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1235271 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 407001600 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 139588270592 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41787085373440 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚41787059200000 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 138243060736 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚41787085373440 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1273827 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1329225216 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41768780558336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0™0Hì%ܽO dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11814554218496 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2518720 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚386403 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚416679 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 139350718778 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1717567488 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1273827 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚819 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1235271 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 407001600 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 139588270592 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41787085373440 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚41787059200000 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 138243060736 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚41787085373440 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1273827 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1329225216 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41768780558336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚819 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41925302260736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚11994142096384 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 139588270592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41926673644032 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚802793 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 284096512 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 138243060736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41925302260736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚41908131277114 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1717567488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49069162496 _step‚9 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49069162496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1994 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7843300930144298e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1994 trainer/global_step‚149 _runtime‚ 253.602093883 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚416390 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1235265 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚819 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1283449 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41768755130368 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1235265 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚803086 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1994 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49069162496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1717567488 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1235271 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚386407 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1994 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 138006119984 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41926673644032 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚11993735094784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41906761250352 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1283449 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38910464 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11814186127360 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 179548967424 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 [HDeviceStatsMonitor.on \Ø’×_train_batch_start/allocated_bytes.large_pool.freed‚41787059200000 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚558 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 179587877888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 HDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2535418 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12072400806400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1717567488 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚819 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49069162496 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1994 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚42046636908544 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1282305 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42065088200704 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42205601513984 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 284096512 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚42046662336512 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38903296 _step‚10 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42065088200704 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406994432 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 139168096256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚149 _runtime‚ 255.041821261 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚419432 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚419142 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚11892042723328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1994 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚42186936521648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1282305 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1243477 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚388943 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚819 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚808085 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1994 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚819 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚558 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚42185566494774 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1717567488 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42065062027264 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 180726174208 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11891674632192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1994 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1329225216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1282305 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1243477 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚388943 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚819 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚808085 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1994 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚819 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚558 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚42185566494774 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1717567488 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42065062027264 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 180726174208 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11891674632192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1994 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1329225216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2535418 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12072400806400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1717567488 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚819 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49069162496 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1994 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚42046636908544 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1282305 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42065088200704 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42205601513984 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 284096512 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚42046662336512 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38903296 _step‚10 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42065088200704 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406994432 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 139168096256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚149 _runtime‚ 255.041821261 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚419432 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚419142 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚11892042723328 2®ãÉ0â+ _wandbruntime‚255  _runtime‚255ìë·bÿãù /loss/FRt1/assignment_loss‚4.568403238771379 /loss/FBt1/detection_loss‚16.578941328512187 .loss/FBt2/detection_loss‚3.493757385513228 2loss/SRqqt2/assignment_loss‚2.1865446983599197 0loss/FRt1/detection_loss‚0.44321939627304235 /loss/FRt2/detection_loss‚1.4022653540829504 0loss/SRqqt1/detection_loss‚4.271040431937278 1loss/SRqqt2/detection_loss‚1.0945960537182757 1loss/FBt1/assignment_loss‚0.24025521507894154 %loss/total_loss‚42.40779317997347 epoch‚0 trainer/global_step‚149 % _timestamp‚1.7843300945081282e+09 _step‚11 .loss/FRt2/assignment_loss‚5.32883064737433 1loss/SRqqt1/assignment_loss‚2.612864255428314 1loss/FBt2/assignment_loss‚0.18707517492361844 _runtime‚ 255.093455095Š*äý .loss/FRt2/assignment_loss‚5.32883064737433 1loss/SRqqt1/assignment_loss‚2.612864255428314 1loss/FBt2/assignment_loss‚0.18707517492361844 _runtime‚ 255.093455095 /loss/FRt1/assignment_loss‚4.568403238771379 /loss/FBt1/detection_loss‚16.578941328512187 .loss/FBt2/detection_loss‚3.493757385513228 2loss/SRqqt2/assignment_loss‚2.1865446983599197 0loss/FRt1/detection_loss‚0.44321939627304235 /loss/FRt2/detection_loss‚1.4022653540829504 0loss/SRqqt1/detection_loss‚4.271040431937278 1loss/SRqqt2/detection_loss‚1.0945960537182757 1loss/FBt1/assignment_loss‚0.24025521507894154 %loss/total_loss‚42.40779317997347 epoch‚0 trainer/global_step‚149 % _timestamp‚1.7843300945081282e+09 _step‚11 G5…Ô“å‚(:ˆ ýîêÒÙß”È gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.75886535644531* gpu.0.memoryAllocatedBytes‚ 49614487552 gpu.0.temp‚43.0 gpu.0.powerWatts‚270.414' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—ÂDÖæ‚(:Ë ýîêÒ›ôÓÊ! disk.dm-0.out‚130.93115234375 disk.nvme0n1.in‚ 327.49609375 disk.dm-1.out‚5.671875 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd2.out‚0 disk.sda1.in‚0) proc.memory.percent‚16.85492992401123 cpu‚25.470441745995526 disk.rbd1.out‚0.75) disk./.usagePercent‚38.98785306715868" disk.nvme0c0n1.in‚ 327.49609375 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚5.671875 disk.drbd2165.out‚0# memory_percent‚86.9373083114624) proc.memory.availableMB‚ 2140.19140625# proc.memory.rssMB‚ 2761.51171875 network.recv‚111503 disk.dm-0.in‚ 327.49609375$ disk.nvme0n1.out‚130.93115234375 disk.sda3.out‚5.671875 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚250986& disk.nvme0c0n1.out‚130.93115234375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0$ disk./.usageGB‚580.9471855163574 disk.drbd2165.in‚0ã»»ç‚(:’ ŒïêÒ¿½ÿÇ gpu.0.gpu‚99.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.75886535644531* gpu.0.memoryAllocatedBytes‚ 49614487552 gpu.0.temp‚43.0 gpu.0.powerWatts‚272.86' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.571999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d]µÍè‚(: ŒïêÒìÈÊ disk.sda1.in‚0 disk.dm-1.in‚0 cpu‚25.467052292686027* disk./.usagePercent‚38.987855627234026! disk.nvme0n1.out‚ 138.57421875 disk.sda3.in‚0 disk.rbd0.in‚0 network.sent‚287981 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda.out‚ 5.93359375# disk.nvme0c0n1.out‚ 138.57421875 disk.rbd2.in‚0 disk.drbd2165.out‚0$ memory_percent‚86.93740367889404) proc.memory.availableMB‚ 2140.17578125 network.recv‚115186 disk.rbd2.out‚0$ disk./.usageGB‚580.9472236633301 disk.rbd1.out‚0.75 disk.nvme0c0n1.in‚ 328.453125 disk.dm-0.out‚ 138.57421875 disk.nvme0n1.in‚ 328.453125 disk.sda3.out‚ 5.93359375 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 328.453125! proc.memory.rssMB‚ 2761.515625 disk.rbd0.out‚0 disk.dm-1.out‚ 5.93359375) proc.memory.percent‚16.85495376586914 proc.cpu.threads‚17‰KVé‚(:’ ›ïêÒ“½ãÇ gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.75886535644531* gpu.0.memoryAllocatedBytes‚ 49614487552 gpu.0.temp‚43.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã0nÛê‚(:Ð ›ïêÒùͱÉ disk.rbd1.out‚0.828125 disk.dm-0.in‚ 404.87109375 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 2140.18359375) proc.memory.percent‚16.85497760772705# disk./.usageGB‚580.947193145752" disk.nvme0c0n1.in‚ 404.87109375 disk.nvme0n1.in‚ 404.87109375 disk.dm-0.out‚ 150.228515625 disk.rbd2.out‚0$ memory_percent‚86.93735599517822 disk.drbd2165.in‚0 disk.sda3.out‚ 6.23046875 cpu‚25.463382875343072 network.recv‚118860 disk.rbd0.in‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2761.51953125 proc.cpu.threads‚17 network.sent‚293381* disk./.usagePercent‚38.987853579173745 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 6.23046875$ disk.nvme0c0n1.out‚ 150.228515625 disk.sda3.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 6.23046875 disk.rbd1.in‚0 disk.drbd2165.out‚0" disk.nvme0n1.out‚ 150.228515625 disk.sda1.in‚0óäC”ë‚(:‰ ªïêÒÔÿ¨Ç gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚253.982' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡cØì‚(:Í ªïêÒžÕ³É network.sent‚298839 disk.nvme0n1.in‚ 404.87109375 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 cpu‚25.461320465126896 disk.sda.in‚0 disk.dm-0.in‚ 404.87109375 disk.dm-0.out‚ 154.958984375 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 2140.1640625 disk.drbd2165.out‚0 disk.sda1.out‚0$ disk./.usageGB‚580.9473342895508 disk.drbd2165.in‚0" disk.nvme0n1.out‚ 154.958984375 disk.sda3.in‚0 disk.rbd2.out‚0 disk.rbd1.out‚0.828125* disk./.usagePercent‚38.987863051452514" disk.nvme0c0n1.in‚ 404.87109375 disk.sda3.out‚ 6.3984375) proc.memory.percent‚16.85497760772705 disk.sda.out‚ 6.3984375$ disk.nvme0c0n1.out‚ 154.958984375 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0$ memory_percent‚86.93747520446777# proc.memory.rssMB‚ 2761.51953125 proc.cpu.threads‚17 network.recv‚122443 disk.dm-1.out‚ 6.3984375³ŒÇ”í‚(:‰ ¹ïêÒËÛâÈ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚278.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×ï2¡Íî‚(: ¹ïêÒÆżÉ disk.rbd2.out‚0 cpu‚25.458720538420746 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.rbd1.out‚0.890625% disk.nvme0c0n1.out‚165.9013671875 disk.rbd0.in‚0) proc.memory.availableMB‚ 2140.16015625) proc.memory.percent‚16.85497760772705 proc.cpu.threads‚17 network.sent‚304364 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda3.out‚6.765625$ memory_percent‚86.93749904632568( disk./.usagePercent‚38.9878584433169 disk.nvme0c0n1.in‚ 405.828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚6.765625 network.recv‚126413 disk./.usageGB‚ 580.947265625 disk.drbd2165.in‚0 disk.dm-0.in‚ 405.828125 disk.nvme0n1.in‚ 405.828125# disk.nvme0n1.out‚165.9013671875# proc.memory.rssMB‚ 2761.51953125 disk.drbd2165.out‚0 disk.dm-0.out‚165.9013671875 disk.sda.out‚6.765625mþõ0ï+ _wandbruntime‚337  _runtime‚337#tÆðÀ ' lr-AdamW/pg1‚0.00024003051463667468 ' lr-AdamW/pg2‚0.00024003051463667468 trainer/global_step‚199 % _timestamp‚1.7843301764888327e+09 _runtime‚ 337.074134552 _step‚12†Žl8ÊñÄ ' lr-AdamW/pg2‚0.00024003051463667468 trainer/global_step‚199 % _timestamp‚1.7843301764888327e+09 _runtime‚ 337.074134552 _step‚12 ' lr-AdamW/pg1‚0.00024003051463667468 ³>0ò+ _wandbruntime‚337  _runtime‚337BËõÓó»O OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1645565 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚306 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15926471601664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚55847087272260 trainer/global_step‚199 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 184558991360 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚513407 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚554749 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55873130937856 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚55871759554560 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 185587831886 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 409098752 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55873130937856 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 185904201216 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1699579 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1645571 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1709201 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55687226736640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 184558991360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚554447 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚ 337.076226195 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚55687226736640 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1699579 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1067850 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1645565 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚513403 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 237901240320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.784330176490536e+09 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1709201 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55871759554560 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3354772 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 _step‚13 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679937024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55848457299022 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15688611368960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 185587831886 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 409098752 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55873130937856 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 185904201216 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1699579 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1645571 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1709201 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55687226736640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 184558991360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚554447 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚ 337.076226195 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚55687226736640 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1699579 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1067850 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1645565 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚513403 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 237901240320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.784330176490536e+09 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1709201 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55871759554560 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3354772 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 _step‚13 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679937024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55848457299022 %#(0õ+ _wandbruntime‚338  _runtime‚338pÇj5Ì&öÆM BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 185166699338 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 186829243904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15766099873792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 338.478072136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚56152058807808 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚16005531921408 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55940725817344 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1653771 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1073414 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚14 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 % _timestamp‚1.7843301778927314e+09 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1717693 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1073116 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚557467 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56127262543556 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1653777 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 186829243904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚41000448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚55940751245312 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1653777 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 186511298244 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3371470 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3361828 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 409091584 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1708057 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚16005122829824käi÷ÊM QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1717693 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚55940751245312 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1653777 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 186511298244 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3371470 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3361828 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 409091584 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1708057 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚16005122829824 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 185166699338 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 186829243904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15766099873792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 338.478072136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚56152058807808 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚16005531921408 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55940725817344 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1653771 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1073414 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚14 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 % _timestamp‚1.7843301778927314e+09 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1717693 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1073116 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚557467 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56127262543556 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1653777 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 186829243904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚41000448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ODeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69818216848384 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1333003 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚69787413294168 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2055871 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚69818216848384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚17 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4190824 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2125331 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404904448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1332733 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 230480346200 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 trainer/global_step‚249 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2055865 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19859826611712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 232220131840 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 $ _timestamp‚1.784330260232382e+09 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4181196 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 232220131840 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69587368099840 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚692330 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚640407 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19859421707264 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19563404701696 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚69819588231680 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 230874921984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚69587341926400 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 296385096704 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚69788783320930 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69587368099840 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2055865 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36813312 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 296421910016 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚ 420.817988296 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2125331 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚69587341926400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192螪Œ¿O ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404904448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1332733 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 230480346200 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 trainer/global_step‚249 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2055865 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19859826611712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 232220131840 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 $ _timestamp‚1.784330260232382e+09 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4181196 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 232220131840 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69587368099840 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚692330 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚640407 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19859421707264 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19563404701696 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚69819588231680 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 230874921984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚69587341926400 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 296385096704 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚69788783320930 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69587368099840 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2055865 NFDeviceSta•²ÀtsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36813312 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 296421910016 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚ 420.817988296 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2125331 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚69587341926400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚640403 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚692596 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19563036610560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚69556932947968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69818216848384 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1333003 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚69787413294168 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2055871 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚69818216848384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚17 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4190824 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2125331 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Çõ,À0+ _wandbruntime‚422  _runtime‚422ìùDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 231799957504 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4207522 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 _step‚18 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 233145174528 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚695083 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2143445 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2064071 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1338026 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚19640525115392 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 232748411352 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2064077 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36806144 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2064071 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69834840154112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7843302616475508e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69865344753664 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 231799957504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚ 422.23323232 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69865370927104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2133809 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69865344753664 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 233145174528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1338294 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70067588565464 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 231403812446 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70098516101632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4197880 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2133809 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚642947ÄÔ_Ï&ÉM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2143445 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2133809 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69865344753664 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 233145174528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1338294 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70067588565464 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 231403812446 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70098516101632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4197880 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2133809 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚642947 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70098516101632 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19938493096448 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 231799957504 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4207522 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 _step‚18 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 233145174528 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚695083 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2143445 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2064071 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1338026 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚19640525115392 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 232748411352 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2064077 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36806144 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2064071 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69834840154112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7843302616475508e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69865344753664 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 231799957504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚ 422.23323232 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69865370927104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚83766045525504 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2466171 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1598309 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34716160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚ 504.184388912 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚83766045525504 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2560705 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1598048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83727739316076 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402807296 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83451047284736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2466165 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚767407 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2466171 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚83487509463040 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚830902 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚767403 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23793650658816 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679937024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 277190852608 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 278536062464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚830645 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 355820715520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚83451021856768 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚299 _step‚21 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚83764674142208 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚23793247851520 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83764674142208 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 278536062464 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2466165 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83764674142208 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 278536062464 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2466165 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚83766045525504 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2466171 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1598309 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34716160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚ 504.184388912 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚83766045525504 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2560705 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1598048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83727739316076 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402807296 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83451047284736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2466165 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚767407 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2466171 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚83487509463040 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚830902 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚767403 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23793650658816 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679937024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0.a0£+ _wandbruntime‚505  _runtime‚505ÛèÎ Ì&¤ÆM I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2474371 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84007914587372 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚833404 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2474377 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1603347 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚833651 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _step‚22 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23514950356992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83728929062912 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5043574 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚83765486116864 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1603598 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84044973395456 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 278985524460 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚84043602004992 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84044973395456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5033932 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83765486116864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5043574 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 278115888128 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚769943 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402800128 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 % _timestamp‚1.7843303450163743e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23515318448128 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2569197 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2474371 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2559561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2474377 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5033932 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚83765512290304 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 278115888128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 279461105152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008ÚsoŠ¥ÊM J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5043574 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 278115888128 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚769943 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402800128 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 % _timestamp‚1.7843303450163743e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23515318448128 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2569197 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2474371 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2559561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2474377 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5033932 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚83765512290304 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 278115888128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 279461105152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2474371 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84007914587372 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚833404 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2474377 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1603347 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚833651 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _step‚22 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23514950356992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83728929062912 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5043574 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚83765486116864 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1603598 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84044973395456 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 278985524460 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚84043602004992 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84044973395456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5033932 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83765486116864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440§|v0¦+ _wandbruntime‚505  _runtime‚505«v]Ÿü§ö 1loss/SRqqt2/assignment_loss‚1.576850252223573 .loss/FRt2/detection_loss‚1.345787789341448 .loss/FBt1/detection_loss‚2.896591024692764 epoch‚0 1loss/FBt1/assignment_loss‚0.20067606084528933 1loss/SRqqt1/detection_loss‚3.1646043737484257 1loss/SRqqt2/detection_loss‚0.9877590058663296 -loss/FBt2/detection_loss‚9.81451400021712 _runtime‚ 505.674335121 /loss/FRt1/assignment_loss‚4.437135347366333 .loss/FRt2/assignment_loss‚4.60207012823557 2loss/SRqqt1/assignment_loss‚2.0022063596070114 0loss/FRt1/detection_loss‚0.35556367249574394 _step‚23 0loss/FBt2/assignment_loss‚0.6214925098970222 %loss/total_loss‚32.00525052453663 trainer/global_step‚299 % _timestamp‚1.7843303450891514e+09FÃ8R¨ú 1loss/SRqqt2/assignment_loss‚1.576850252223573 .loss/FRt2/detection_loss‚1.345787789341448 .loss/FBt1/detection_loss‚2.896591024692764 epoch‚0 1loss/FBt1/assignment_loss‚0.20067606084528933 1loss/SRqqt1/detection_loss‚3.1646043737484257 1loss/SRqqt2/detection_loss‚0.9877590058663296 -loss/FBt2/detection_loss‚9.81451400021712 _runtime‚ 505.674335121 /loss/FRt1/assignment_loss‚4.437135347366333 .loss/FRt2/assignment_loss‚4.60207012823557 2loss/SRqqt1/assignment_loss‚2.0022063596070114 0loss/FRt1/detection_loss‚0.35556367249574394 _step‚23 0loss/FBt2/assignment_loss‚0.6214925098970222 %loss/total_loss‚32.00525052453663 trainer/global_step‚299 % _timestamp‚1.7843303450891514e+09=ÜA&”©‚(:‰ íðêÒ¡è†Ê gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢ÍN–ં(:Õ íðêÒ‡ñÜË disk.dm-1.in‚0 cpu‚25.43221108958696 disk.dm-0.in‚ 641.03515625 disk.sda1.out‚0 disk.rbd2.out‚0 disk.drbd2165.out‚0$ memory_percent‚86.94055080413818 proc.cpu.threads‚17 disk.dm-1.out‚ 10.21484375 disk.sda.in‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 641.03515625 network.recv‚173955) proc.memory.percent‚16.85507297515869$ disk./.usageGB‚580.9473991394043! disk.dm-0.out‚253.40673828125 disk.sda3.in‚0# proc.memory.rssMB‚ 2761.53515625 disk.sda3.out‚ 10.21484375 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 10.21484375 network.sent‚466294 disk.rbd2.in‚0 disk.rbd1.out‚1.3125) proc.memory.availableMB‚ 2139.66015625 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚253.40673828125( disk./.usagePercent‚38.9878674035806 disk.nvme0n1.in‚ 641.03515625$ disk.nvme0n1.out‚253.40673828125 disk.sda1.in‚0‘k„Óž«‚(:“ üðêÒÈáéÇ gpu.0.gpu‚83.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å¯hžá¬‚(:Ö üðêÒ›öÉ disk.sda2.out‚0 disk.rbd1.out‚1.3125 disk.sda.out‚ 10.45703125 disk.dm-0.in‚ 641.03515625$ memory_percent‚86.94024085998535! disk.dm-0.out‚257.40576171875) disk./.usagePercent‚38.98786151540731 disk.nvme0n1.in‚ 641.03515625& disk.nvme0c0n1.out‚257.40576171875# proc.memory.rssMB‚ 2761.53515625 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.drbd2165.in‚0 cpu‚25.430595729495305 disk.dm-1.in‚0 disk.dm-1.out‚ 10.45703125 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd2.out‚0 network.sent‚500948$ disk.nvme0n1.out‚257.40576171875 disk.sda3.out‚ 10.45703125 disk.rbd1.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.recv‚178343$ disk./.usageGB‚580.9473114013672 disk.sda1.out‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 641.03515625( proc.memory.availableMB‚ 2139.7109375) proc.memory.percent‚16.85507297515869 n¿T”­‚(:‰ ‹ñêÒºÍÕÈ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚248.859' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.7718 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óP͌ޮ‚(:Ó ‹ñêÒ¢ó‹É disk.sda1.out‚0 disk.sda3.out‚ 10.734375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0& disk.nvme0c0n1.out‚266.66455078125) proc.memory.percent‚16.85507297515869$ disk.nvme0n1.out‚266.66455078125 disk.dm-1.out‚ 10.734375 disk.sda.in‚0 disk.sda.out‚ 10.734375 disk.dm-0.in‚ 725.03515625$ memory_percent‚86.94031238555908) disk./.usagePercent‚38.98786689156553 disk.sda3.in‚0# proc.memory.rssMB‚ 2761.53515625 proc.cpu.threads‚17 network.recv‚182057$ disk./.usageGB‚580.9473915100098 disk.rbd0.in‚0 disk.rbd2.out‚0 disk.rbd1.out‚1.390625 network.sent‚506427 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0! disk.dm-0.out‚266.66455078125 cpu‚25.428534883638502 disk.nvme0n1.in‚ 725.03515625 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 725.03515625) proc.memory.availableMB‚ 2139.69921875P;J;“¯‚(:ˆ šñêÒ™”ýÈ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚252.925' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0- óðÞ°‚(:Ó šñêÒÊÓƒÊ proc.cpu.threads‚17 disk.rbd1.out‚1.390625 disk.drbd2165.out‚0$ disk./.usageGB‚580.9478492736816 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚86.94188594818115) proc.memory.percent‚16.85507297515869 disk.sda1.out‚0 disk.rbd2.out‚0 disk.sda.in‚0 cpu‚25.427280663221502 disk.rbd1.in‚0 network.recv‚185299) disk./.usagePercent‚38.98789761246964 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 10.8828125 disk.drbd2165.in‚0! disk.dm-0.out‚274.09326171875 disk.rbd2.in‚0 disk.sda3.out‚ 10.8828125! disk.nvme0c0n1.in‚ 725.9921875 disk.dm-0.in‚ 725.9921875) proc.memory.availableMB‚ 2139.44140625# proc.memory.rssMB‚ 2761.53515625 disk.sda.out‚ 10.8828125 network.sent‚511806& disk.nvme0c0n1.out‚274.09326171875 disk.nvme0n1.in‚ 725.9921875$ disk.nvme0n1.out‚274.09326171875 disk.sda3.in‚0 disk.dm-1.in‚0Š;|Ç”±‚(:‰ ©ñêÒÃí«Ç gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚324.933' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.9866 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áÑ[DÞ²‚(:Ó ©ñêÒÖÜ­É disk.rbd2.in‚0 disk.rbd2.out‚0$ disk./.usageGB‚580.9478416442871 disk.sda2.out‚0 disk.dm-0.in‚ 725.9921875 disk.rbd1.in‚0 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚280.34912109375 disk.sda.out‚ 11.23046875% proc.memory.availableMB‚ 2139.9375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.in‚0! disk.dm-0.out‚280.34912109375 proc.cpu.threads‚17 network.sent‚516145 disk.nvme0n1.in‚ 725.9921875 disk.rbd0.in‚0 disk.dm-1.in‚0) proc.memory.percent‚16.85507297515869 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 725.9921875$ memory_percent‚86.93885803222656 network.recv‚188502$ disk.nvme0n1.out‚280.34912109375 disk.sda1.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2761.53515625* disk./.usagePercent‚38.987897100454575 disk.sda1.out‚0 disk.sda3.out‚ 11.23046875 disk.dm-1.out‚ 11.23046875 disk.rbd1.out‚1.515625 cpu‚25.425812914500654oÂ’l”³‚(:‰ ¸ñêÒà¯ÓÇ gpu.0.gpu‚54.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚326.524' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.3048 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZV¦Ü´‚(:Ñ ¸ñêÒù—ÄÉ# disk./.usageGB‚580.947826385498 disk.rbd2.out‚0 disk.drbd2165.in‚0# memory_percent‚86.9368314743042 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda.out‚ 13.015625" proc.memory.rssMB‚ 2761.5390625* disk./.usagePercent‚38.987896076424434 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0! disk.dm-0.out‚292.52001953125 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda.in‚0 cpu‚25.424345630259232 network.sent‚522074 disk.nvme0n1.in‚ 726.82421875$ disk.nvme0n1.out‚292.52001953125 disk.rbd1.out‚ 1.5234375( proc.memory.percent‚16.8550968170166 disk.sda3.in‚0& disk.nvme0c0n1.out‚292.52001953125 network.recv‚192992 disk.sda3.out‚ 13.015625 disk.sda2.out‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 726.82421875) proc.memory.availableMB‚ 2140.26953125 disk.sda2.in‚0 disk.dm-1.out‚ 13.015625 disk.dm-0.in‚ 726.82421875% Jð0µ+ _wandbruntime‚588  _runtime‚588÷ØÔƶÀ _runtime‚ 588.021919833 _step‚24 ' lr-AdamW/pg1‚0.00042095803823215814 ' lr-AdamW/pg2‚0.00042095803823215814 trainer/global_step‚349 % _timestamp‚1.7843304274367628e+09p¯pšÊ·Ä _step‚24 ' lr-AdamW/pg1‚0.00042095803823215814 ' lr-AdamW/pg2‚0.00042095803823215814 trainer/global_step‚349 % _timestamp‚1.7843304274367628e+09 _runtime‚ 588.021919833?}Œ0¸+ _wandbruntime‚588  _runtime‚588mÊEÀ'¹ºO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2876471 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚894403 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27727839008256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97345136193536 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2876471 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _step‚25 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚97712502819328 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 415551204352 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 324851993088 $ _timestamp‚1.784330427438233e+09 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2976835 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 324851993088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚97711131436032 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚969855 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400710144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97387650826240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 323506783232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2986457 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97387624652800 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 322954572416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5862928 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97387624652800 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚97712502819328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚97668065337984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97387650826240uá!‰º¾O VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 323506783232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2986457 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97387624652800 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 322954572416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5862928 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97387624652800 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚97712502819328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚97668065337984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97387650826240 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2876471 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚894403 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27727839008256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97345136193536 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2876471 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _step‚25 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚97712502819328 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 415551204352 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.sma›‹×ôCll_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 324851993088 $ _timestamp‚1.784330427438233e+09 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2976835 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 324851993088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚97711131436032 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚969855 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400710144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97387650826240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175^Ðâ0»+ _wandbruntime‚589  _runtime‚589q@JÌ&¼ÆM M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97948240609280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2985313 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5869984 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 323878038662 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2994949 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97665627480064 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚97991430689280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97623017971712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2994949 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 325777035776 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 324431818752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1869344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 % _timestamp‚1.7843304288552403e+09 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚97990059298816 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚972633 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚896943 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2985313 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 325777035776 _runtime‚ 589.440764493 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚97991430689280 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 416778785280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 trainer/global_step‚349 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5879626 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚97946870582406 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚972401 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 325222637568 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _step‚26 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5879626 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1869580 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚972633 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚896943 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2985313 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 325777035776 _runtime‚ 589.440764493 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚97991430689280 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 416778785280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 trainer/global_step‚349 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5879626 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚97946870582406 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚972401 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 325222637568 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _step‚26 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5879626 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1869580 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97948240609280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2985313 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5869984 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 323878038662 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2994949 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97665627480064 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚97991430689280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97623017971712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2994949 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 325777035776 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 324431818752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1869344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 % _timestamp‚1.7843304288552403e+09 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚97990059298816 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872îð0¾+ _wandbruntime‚589  _runtime‚589K—Ѿý¿÷ /loss/FRt1/assignment_loss‚4.102067160417662 2loss/SRqqt1/assignment_loss‚1.8576717864959797 .loss/FRt2/detection_loss‚1.339682558576091 0loss/SRqqt1/detection_loss‚4.390401277695312 -loss/FBt1/detection_loss‚4.17811364599071 _step‚27 1loss/FBt1/assignment_loss‚0.20694586721323632 _runtime‚ 589.508017222 2loss/SRqqt2/assignment_loss‚1.5621927011973913 0loss/FBt2/assignment_loss‚0.4048369929939508 0loss/FRt1/detection_loss‚0.37363562263723993 1loss/SRqqt2/detection_loss‚0.8851678135552061 .loss/FBt2/detection_loss‚28.81120613892618 %loss/total_loss‚52.13746930231156 % _timestamp‚1.7843304289227474e+09 /loss/FRt2/assignment_loss‚4.025547736612606 epoch‚0 trainer/global_step‚349]›ò,Àû % _timestamp‚1.7843304289227474e+09 /loss/FRt2/assignment_loss‚4.025547736612606 epoch‚0 trainer/global_step‚349 /loss/FRt1/assignment_loss‚4.102067160417662 2loss/SRqqt1/assignment_loss‚1.8576717864959797 .loss/FRt2/detection_loss‚1.339682558576091 0loss/SRqqt1/detection_loss‚4.390401277695312 -loss/FBt1/detection_loss‚4.17811364599071 _step‚27 1loss/FBt1/assignment_loss‚0.20694586721323632 _runtime‚ 589.508017222 2loss/SRqqt2/assignment_loss‚1.5621927011973913 0loss/FBt2/assignment_loss‚0.4048369929939508 0loss/FRt1/detection_loss‚0.37363562263723993 1loss/SRqqt2/detection_loss‚0.8851678135552061 .loss/FBt2/detection_loss‚28.81120613892618 %loss/total_loss‚52.13746930231156Yà1DÁ‚(:… ÇñêÒÐ×íÇ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üã‚(:Ø ÇñêÒÐõÃÉ disk.sda.out‚ 13.57421875* disk./.usagePercent‚38.987925005275805 disk.rbd1.in‚0! disk.dm-0.out‚300.53662109375 disk.rbd0.in‚0 disk.rbd2.out‚0& disk.nvme0c0n1.out‚300.53662109375 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚580.9482574462891 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 881.67578125 disk.sda3.out‚ 13.57421875 disk.rbd2.in‚0) proc.memory.availableMB‚ 2140.03515625( proc.memory.percent‚16.8550968170166 disk.dm-1.out‚ 13.57421875 disk.rbd1.out‚ 1.6015625" proc.memory.rssMB‚ 2761.5390625 proc.cpu.threads‚17$ disk.nvme0n1.out‚300.53662109375 cpu‚25.42336033951662 disk.drbd2165.in‚0 network.sent‚527504 network.recv‚197152 disk.dm-1.in‚0 disk.dm-0.in‚ 881.67578125 disk.sda2.out‚0$ memory_percent‚86.93826198577881" disk.nvme0c0n1.in‚ 881.67578125 disk.sda1.in‚0´¡-”Â(:‰ ÖñêÒ€‚É gpu.0.gpu‚58.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yP¸lÚÄ‚(:Ï ÖñêÒ÷ÉÊ disk.sda3.out‚ 13.8984375 disk.rbd0.out‚0$ disk./.usageGB‚580.9481925964355 disk.sda.in‚0 disk.sda1.in‚0$ memory_percent‚86.93680763244629 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.sda2.out‚0 cpu‚25.421163502670282 disk.sda.out‚ 13.8984375 disk.dm-1.in‚0 network.recv‚202365! disk.nvme0c0n1.in‚ 882.6328125 disk.rbd0.in‚0" proc.memory.rssMB‚ 2761.5390625 network.sent‚575706 disk.dm-0.out‚309.3271484375( proc.memory.availableMB‚ 2140.2734375* disk./.usagePercent‚38.987920653147725 disk.drbd2165.in‚0% disk.nvme0c0n1.out‚309.3271484375 disk.sda2.in‚0 disk.rbd1.out‚ 1.6015625( proc.memory.percent‚16.8550968170166 disk.dm-0.in‚ 882.6328125 disk.nvme0n1.in‚ 882.6328125 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.dm-1.out‚ 13.8984375 disk.rbd1.in‚0 proc.cpu.threads‚17# disk.nvme0n1.out‚309.3271484375 disk.sda1.out‚0Q‘ “”Å‚(:‰ åñêÒ¼öÕÇ gpu.0.gpu‚82.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚111.883' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3766 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëF?«ÞÆ‚(:Ó åñêÒç™ÒÉ disk.sda.in‚0 disk.nvme0n1.in‚ 883.46484375 disk.dm-1.out‚ 14.2109375 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 883.46484375! disk.dm-0.out‚320.94091796875& disk.nvme0c0n1.out‚320.94091796875 disk.sda1.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 2140.203125 disk.drbd2165.out‚0 disk.rbd0.in‚0( proc.memory.percent‚16.8550968170166$ disk.nvme0n1.out‚320.94091796875 disk.sda3.out‚ 14.2109375 disk.dm-1.in‚0" proc.memory.rssMB‚ 2761.5390625 proc.cpu.threads‚17 network.recv‚205908* disk./.usagePercent‚38.987920653147725 disk.sda.out‚ 14.2109375 disk.sda3.in‚0 disk.rbd2.in‚0$ memory_percent‚86.93723678588867 cpu‚25.41999186915771 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd2.out‚0 disk.dm-0.in‚ 883.46484375 disk.rbd1.out‚ 1.6953125 network.sent‚580633$ disk./.usageGB‚580.9481925964355 `ép”Ç‚(:‰ ôñêÒ”©ßÈ gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚117.447' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4894 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vP:sÈ‚(:Ò ôñêÒóγÉ disk.dm-0.in‚ 883.46484375$ memory_percent‚86.93876266479492 cpu‚25.41919060745179! disk.dm-0.out‚325.11474609375$ disk.nvme0n1.out‚325.11474609375 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 883.46484375 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0) disk./.usagePercent‚38.98793089344909 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.sda2.in‚0 disk.sda.out‚ 14.5859375 disk.drbd2165.out‚0( proc.memory.percent‚16.8550968170166 proc.cpu.threads‚17& disk.nvme0c0n1.out‚325.11474609375" disk.nvmeè¿‘Jj0c0n1.in‚ 883.46484375 network.recv‚209252$ disk./.usageGB‚580.9483451843262 disk.sda3.out‚ 14.5859375 network.sent‚585771 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 14.5859375 disk.rbd1.out‚ 1.6953125 disk.sda1.in‚0 disk.drbd2165.in‚0' proc.memory.availableMB‚ 2139.953125" proc.memory.rssMB‚ 2761.5390625žò#ýÉ‚(:’ ƒòêÒñ‡ÑÉ gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.164599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥´ÝnáÊ‚(:Ö ƒòêÒò–àË" proc.memory.rssMB‚ 2761.5390625! disk.dm-0.out‚333.57958984375 disk.nvme0n1.in‚ 948.7109375$ disk.nvme0n1.out‚333.57958984375 disk.sda3.in‚0 disk.rbd2.out‚0 disk.dm-1.out‚ 14.76953125 disk.rbd1.in‚0& disk.nvme0c0n1.out‚333.57958984375( proc.memory.percent‚16.8550968170166 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 948.7109375 network.recv‚212901 disk.drbd2165.out‚0$ memory_percent‚86.93902492523193$ disk./.usageGB‚580.9483261108398 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 14.76953125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd1.out‚ 1.7734375! disk.nvme0c0n1.in‚ 948.7109375) proc.memory.availableMB‚ 2139.91015625 cpu‚25.418587608962653 proc.cpu.threads‚17 network.sent‚592675 disk.rbd0.out‚0* disk./.usagePercent‚38.987929613411424 disk.sda3.out‚ 14.76953125 disk.sda2.in‚0ÖJ?c0Ë+ _wandbruntime‚671  _runtime‚671ãú„kÄ̾ % _timestamp‚1.7843305111655908e+09 _runtime‚ 671.750858429 _step‚28 & lr-AdamW/pg1‚0.0004812672127639859 & lr-AdamW/pg2‚0.0004812672127639859 trainer/global_step‚399nÏ¢ÈÍ & lr-AdamW/pg2‚0.0004812672127639859 trainer/global_step‚399 % _timestamp‚1.7843305111655908e+09 _runtime‚ 671.750858429 _step‚28 & lr-AdamW/pg1‚0.0004812672127639859fjÓt0Î+ _wandbruntime‚671  _runtime‚671j“|MÒ'ÏÌO OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3286765 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚111239225102336 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679937024 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1109086 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 369191685524 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3412209 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2130489 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111657588729856 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 475549221888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 369822713856 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31662262267392 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1021403 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3286771 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1021407 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31661861557248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111609761386654 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚399 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 371167923712 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3286771 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 369191685524 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3412209 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2130489 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111657588729856 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 475549221888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 369822713856 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31662262267392 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1021403 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3286771 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1021407 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31661861557248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111609761386654 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚399 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 371167923712 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3286771 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111517081452544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3420701 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6715678 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111937887983104 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 370747749376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3294971 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34708992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111887196604314 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2136058 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3411065 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3411065 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31264168931328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31263800840192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6706036 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3294971 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 476738046976 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111888566631188 % _timestamp‚1.7843305125829082e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3294977 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 370747749376 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111565768843264 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31740538887168 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111517106880512 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6715678 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1112111 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚399 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2135812 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 673.168764476 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3420701 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚30 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 372092966400 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 372092966400KFœ$ÓÚM E=DeviceStatsMonitor.on_traiçaG¼&n_batch_end/inactive_split.all.peak‚572 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6706036 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3294971 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 476738046976 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111888566631188 % _timestamp‚1.7843305125829082e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3294977 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 370747749376 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111565768843264 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31740538887168 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111517106880512 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6715678 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1112111 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚399 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2135812 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 673.168764476 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3420701 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚30 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 372092966400 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 372092966400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111937887983104 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111517081452544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3420701 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6715678 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111937887983104 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 370747749376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3294971 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34708992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111887196604314 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2136058 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3411065 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3411065 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31264168931328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31263800840192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008Îå”à0Ô+ _wandbruntime‚673  _runtime‚673醽üÕö _runtime‚ 673.23949955 epoch‚0 1loss/SRqqt1/assignment_loss‚2.068246525083039 0loss/FBt2/assignment_loss‚0.3389707241881767 /loss/FRt1/detection_loss‚0.4306913866068953 % _timestamp‚1.7843305126541011e+09 _step‚31 /loss/FRt2/assignment_loss‚3.967632829590349 2loss/SRqqt2/assignment_loss‚1.5554045619375734 /loss/FBt1/detection_loss‚2.3594340766551936 /loss/FRt1/assignment_loss‚4.042191037400202 0loss/FBt1/assignment_loss‚0.1277838395912598 /loss/FRt2/detection_loss‚1.2396259429656427 .loss/FBt2/detection_loss‚6.863916183340138 %loss/total_loss‚28.31693463866933 trainer/global_step‚399 0loss/SRqqt1/detection_loss‚4.396315041426066 1loss/SRqqt2/detection_loss‚0.9267224898847964mͼÖú /loss/FRt2/detection_loss‚1.2396259429656427 .loss/FBt2/detection_loss‚6.863916183340138 %loss/total_loss‚28.31693463866933 trainer/global_step‚399 0loss/SRqqt1/detection_loss‚4.396315041426066 1loss/SRqqt2/detection_loss‚0.9267224898847964 _runtime‚ 673.23949955 epoch‚0 1loss/SRqqt1/assignment_loss‚2.068246525083039 0loss/FBt2/assignment_loss‚0.3389707241881767 /loss/FRt1/detection_loss‚0.4306913866068953 % _timestamp‚1.7843305126541011e+09 _step‚31 /loss/FRt2/assignment_loss‚3.967632829590349 2loss/SRqqt2/assignment_loss‚1.5554045619375734 /loss/FBt1/detection_loss‚2.3594340766551936 /loss/FRt1/assignment_loss‚4.042191037400202 0loss/FBt1/assignment_loss‚0.1277838395912598| çH’ׂ(:‡ ’òêÒŒ—“È gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚234.589' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.9178 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#Û€ÖØ‚(:Ë ’òêÒ›‡ÈÊ$ disk.nvme0n1.out‚343.89501953125 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 957.09375* proc.memory.percent‚16.855788230895996 disk.sda3.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2761.65234375$ disk./.usageGB‚580.9483413696289 disk.sda1.out‚0 disk.sda3.out‚ 15.109375 disk.rbd2.in‚0& disk.nvme0c0n1.out‚343.89501953125$ memory_percent‚86.93957328796387 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.out‚ 15.109375 disk.rbd1.out‚ 1.7734375 network.recv‚216354 disk.sda1.in‚0 network.sent‚595705) disk./.usagePercent‚38.98793063744156 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 957.09375 disk.sda.out‚ 15.109375 disk.nvme0c0n1.in‚ 957.09375( proc.memory.availableMB‚ 2139.8203125 disk.rbd2.out‚0 disk.dm-1.in‚0! disk.dm-0.out‚343.89501953125 cpu‚25.422207745888425ÚXÁ’Ù‚(:‡ ¡òêÒЄžÇ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œÊ¨dÐÚ‚(:Å ¡òêÒ‘ÁêÉ$ disk.nvme0n1.out‚347.54541015625 disk.sda3.in‚0" proc.memory.rssMB‚ 2761.6484375 proc.cpu.threads‚17! disk.dm-0.out‚347.54541015625 disk.rbd1.in‚0 network.recv‚220738 disk.sda.out‚15.40625& disk.nvme0c0n1.out‚347.54541015625 disk.sda1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚16.855764389038086 network.sent‚643011 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 957.09375 disk.rbd2.in‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 957.09375 disk.rbd2.out‚0 disk.rbd1.out‚1.90625) proc.memory.availableMB‚ 2139.78515625$ disk./.usageGB‚580.9486923217773 disk.sda1.out‚0 disk.rbd0.out‚0 cpu‚25.47353007285602) disk./.usagePercent‚38.98795419013471 disk.dm-0.in‚ 957.09375 disk.sda3.out‚15.40625$ memory_percent‚86.93978786468506 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚15.40625ëo–ŒHÛbC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP¬awNÜbI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP¶§ÒQJÝbE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPÔÀZJÞbE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPcÝÞùJßbE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPzt‰ÉJàbE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP>}³ÞJábE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPeUU;âb6 jet/accuracy_2_of_2"trainer/global_step2:JPð:ù?NãbI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP«Ñ˜@JäbE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP9\›HåbC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPK4ÄJæbE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP¥e=HçbC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP}ºüNèbI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPFIرHébC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP ê zJêbE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPæ‡1NëbI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPí1¼HìbC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPU8ì^HíbC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP¢’åëJîbE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPoGmóJïbE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPú‰JðbE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP„£ù7NñbI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPLã;òb6 jet/accuracy_3_of_3"trainer/global_step2:JP‰(ß@ób; particle/accuracy_3_of_3"trainer/global_step2:JPÓ"æ¦;ôb6 jet/accuracy_3_of_5"trainer/global_step2:JP·š»HõbC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP§QNÝNöbI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPw>^ŒH÷bC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPÆ#jHNøbI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPvç=o@ùb; particle/accuracy_1_of_2"trainer/global_step2:JPô©÷Ø@úb; particle/accuracy_3_of_4"trainer/global_step2:JPW´›NûbI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPzôNübI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPäøéuHýbC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP—Á6•HþbC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP˜°ÐRJÿbE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP‰S€N€bI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP½ª«NbI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP è›;‚b6 jet/accuracy_1_of_1"trainer/global_step2:JPwsØ2HƒbC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP,Y0²J„bE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPa#ç H…bC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP§ÉÖùJ†bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPšôSÈJ‡bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP”¢Á“8ˆb3 particle/f_score"trainer/global_step2:JP²?`bH‰bC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP*G‘NŠbI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP‡úJ‹bE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPõQCHŒbC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP,ÍY¨HbC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPÆÚK;Žb6 loss/val_total_loss"trainer/global_step2:JPVŠ„ôHbC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPó‰ÂHbC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPB¸X•N‘bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPGô+gN’bI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPùµWH“bC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP ´¬N”bI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPQðlÌJ•bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP™#†ÆJ–bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP5»eOJ—bE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPÂAWUJ˜bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPþ÷ž·N™bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPû»íENšbI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPê>³?N›bI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP3Uè¾JœbE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPÝã@b; particle/accuracy_1_of_1"trainer/global_step2:JPÕZÀJžbE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP`OINŸbI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPAÈ4J bE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPëäðˆH¡bC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP$X|H¢bC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPT§ûJ£bE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP¤^%;¤b6 jet/accuracy_1_of_4"trainer/global_step2:JPZ|€J¥bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÀX òH¦bC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP„¯!;J§bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP”hJ¨bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP†x±N©bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP€ÌÐ.JªbE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÃCz J«bE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPaS‘N¬bI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPÐ ÝAN­bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÞãt[H®bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP<  J¯bE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP„´|J°bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPÆØe]H±bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPqܧ½H²bC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP‚FÀJ³bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPE„îN´bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPv#6HµbC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPnñz8J¶bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPj²Xz;·b6 jet/accuracy_2_of_5"trainer/global_step2:JP °˜\N¸bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP¸KÚíJ¹bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPùt@NºbI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP®n|ÂH»bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPДmJ¼bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPþŸ&H½bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP[ÁÊÍH¾bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPæTMH¿bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP·K)¨JÀbE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP] PNÁbI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP¥‘u×@Âb; particle/accuracy_2_of_4"trainer/global_step2:JPðhJÃbE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÚ‘óòJÄbE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPá¢àJÅbE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP’+íJÆbE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP× qt@Çb; particle/accuracy_3_of_5"trainer/global_step2:JP6#ÊüNÈbI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP`Å'1@Éb; particle/accuracy_2_of_3"trainer/global_step2:JPÓ¨¥YHÊbC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPji]ÛHËbC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP ÆÉNÌbI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÊg›JÍbE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÓÌúJÎbE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPˆ$ð`JÏbE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPÄ¿@¸HÐbC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP¹Ò2^JÑbE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPaô LNÒbI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP3³è¸JÓbE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP?>¯»JÔbE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP:σ,HÕbC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP}ÑbHÖbC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP°:àJ×bE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP»î,áJØbE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP&‡;ÖNÙbI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPCÊí¸NÚbI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP{aµ»NÛbI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPbéˆNHÜbC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPõVNNÝbI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP@ÁFNÞbI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPÚx™JßbE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPŠ®JàbE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPs\)‡HábC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPÛlôëHâbC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPÎl»JãbE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPÖ·­JäbE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPb±}NåbI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPZH°ôJæbE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPÜõTnJçbE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP¦¹6°NèbI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP‹t€¸JébE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÙ¾9NêbI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP¤›¥¼;ëb6 jet/accuracy_4_of_5"trainer/global_step2:JPÓЧHìbC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPìQ (HíbC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP‡€ýZJîbE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPCo‚HïbC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP8pêâJðbE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPfÚ¡JñbE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPòeó»JòbE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP‡•  ;ób6 jet/accuracy_1_of_2"trainer/global_step2:JP>íÖHôbC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPÞ|×+JõbE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPôÒtNöbI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPs*36J÷bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP¡»·W<øb7 particle/specificity"trainer/global_step2:JP Ã-~HùbC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP³?,´JúbE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP¡ÐHûbC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPý\¬KJübE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÓÈô?JýbE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP>ëcòNþbI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPçŠOO;ÿb6 jet/accuracy_1_of_3"trainer/global_step2:JPeù³þJ€bE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP…¬lJbE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP€)íêN‚bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPñîdŸNƒbI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPD´¯^J„bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP„Æ‚€J…bE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPÖ¬¡J†bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP +«“@‡b; particle/accuracy_5_of_5"trainer/global_step2:JPïø^ÁNˆbI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPâjKÔJ‰bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPŽN;Šb6 jet/accuracy_3_of_4"trainer/global_step2:JP€ˆ@‹b; particle/accuracy_4_of_4"trainer/global_step2:JPDÞmJŒbE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPmµáaJbE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP—ö~JŽbE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPf» ”NbI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPû_?†HbC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPGÎÞJ‘bE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÏx#H’bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPÌKŽZJ“bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP+3%åN”bI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP>¾®HJ•bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÙ¦‚mH–bC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÍ ¶J—bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÔÐ"V@˜b; particle/accuracy_4_of_5"trainer/global_step2:JPN>']N™bI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÊ«:)HšbC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPÓ@J›bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPW£äbNœbI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPu4ñ0@b; particle/accuracy_1_of_5"trainer/global_step2:JPê—¿@žb; particle/accuracy_2_of_5"trainer/global_step2:JP€‰HŸbC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPl´òÉ9 b4 particle/accuracy"trainer/global_step2:JPßø¨žH¡bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPý|é¾J¢bE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPkåȺH£bC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP¼ösN¤bI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÈžúsH¥bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPA»0,J¦bE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP(¤ŸJ§bE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPðj~¥N¨bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPüÙþH©bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP›ÍxJªbE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPøV6GJ«bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP(aé†N¬bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP+V´J­bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPœ/Š’J®bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPo^‡ÈN¯bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP>OŠTH°bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP_ès@±b; particle/accuracy_1_of_4"trainer/global_step2:JPÅú¥jH²bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP'xxåH³bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPÂ¥zÊH´bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP¨ HµbC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP‰ä–§H¶bC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPÕ©wlH·bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÇB‘¤H¸bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPgàìJ¹bE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP˜˜jžHºbC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPz·H»bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP>Gf¢N¼bI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP¼T¬N½bI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPnÀSN¾bI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPP’6J¿bE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP¯0qWHÀbC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÒ ÍC;Áb6 jet/accuracy_2_of_3"trainer/global_step2:JP~}VŽ Âb6 je:² 2t/accuracy_4_of_4"trainer/global_step2:JP¤š#aGÃbB validation_average_jet_accuracy"trainer/global_step2:JPd9JÄbE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPuÊcZ;Åb6 jet/accuracy_5_of_5"trainer/global_step2:JPÓ}HÆbC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP¶áNÇbI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPVÜ·JÈbE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP¯œÎ¦HÉbC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP¶YœÓJÊbE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPo8 ‘HËbC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPYi»NÌbI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPÚ”Í@Íb; particle/accuracy_1_of_3"trainer/global_step2:JP-9*eJÎbE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP­'›ŠHÏbC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPóQÍHÐbC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPW+XãNÑbI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP^U©§HÒbC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP®¤JÓbE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP_Œ¾±NÔbI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP»£:NÕbI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPW„¾(@Öb; particle/accuracy_2_of_2"trainer/global_step2:JPÅãTûH×bC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP­±œ²JØbE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP'oxNÙbI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPÖ›ÈHÚbC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPebv¸JÛbE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP#:J;Üb6 jet/accuracy_2_of_4"trainer/global_step2:JP¹wóNÝbI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP–|†HÞbC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP|ŽˆHßbC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPpy]²JàbE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPÍÇ5NábI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP÷_È÷NâbI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPj2ªHãbC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP1GñJäbE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPÓžÑNHåbC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPtbûHæbC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP¸°HçbC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPû4HèbC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP›R»HébC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP¨ë…JêbE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP\î @JëbE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP™ËóJìbE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP”,‹ºHíbC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPÊIÝHîbC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPûH”RHïbC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPØÉÁMJðbE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPõÊl=NñbI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPSyƒrHòbC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP ~béHóbC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPÙØ h;ôb6 jet/accuracy_1_of_5"trainer/global_step2:JPÔ‡ $HõbC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPÿU¯=NöbI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP¾IµòJ÷bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP×Ç.~<øb7 particle/sensitivity"trainer/global_step2:JP¶§à¿HùbC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP ÏzrJúbE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP?7ˆë0û+ _wandbruntime‚701  _runtime‚701¼-î=ü•z >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9493302702903748 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9968253374099731 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.08131343126296997 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.016203705221414566 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.10745243728160858 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9767648577690125 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9931907653808594 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.03956804424524307 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_2_of_4‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.07706161588430405 ;"Purity/1FRt*SRqqt1FBt/event_purity‚0.010296827182173729 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.14944547414779663 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_3_of_5‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9900862574577332 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.07653003931045532 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.14351852238178253 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.07120181620121002 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.016841910779476166 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.0624452643096447 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9841940999031067 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.094841368496418 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11045478284358978 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0011614401591941714 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08270248025655746 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.06925841420888901 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.2770010232925415 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.10406067222356796 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9963717460632324 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0012954147532582283 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7477262616157532 <"Purity/0FRt2SRqqt*FBt/event_purity‚0.0031704306602478027 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.0748816579580307 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9943122863769531 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.1536659151315689 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08229292929172516 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7181670665740967 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.1917344331741333 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.20000000298023224 +jet/accuracy_1_of_2‚0.29250505566596985 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7486076354980469 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.012626002542674541 +particle/specificity‚0.7571113109588623 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.07176071405410767 ="Purity/1FRt1SRqqt0FBt/event_purity‚0.00046650946023873985 ) Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 _step‚32 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.4901600480079651 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.08842748403549194 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0006349206669256091 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_3_of_4‚0.002801120514050126 /particle/accuracy_4_of_4‚0.9893046021461487 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.2167254388332367 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.08706612139940262 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.12081970274448395 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9867726564407349 ="Purity/1FRt1SRqqt*FBt/event_purity‚0.00044689406058751047 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.01892898604273796 ;"Purity/*FRt*SRqqt0FBt/event_purity‚0.025126900523900986 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.1689140498638153 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11234768480062485 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.10156447440385818 particle/accuracy_4_of_5‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10986337065696716 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.28240740299224854 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08270248025655746 (particle/accuracy‚0.7121155858039856 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9963901042938232 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.12857992947101593 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9899304509162903 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.017089461907744408 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.12081970274448395 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.05337870866060257 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.10821931809186935 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.09116145223379135 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9924991130828857 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.10169368982315063 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.993740975856781 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.30000001192092896 ) Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.09263042360544205 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.08086773008108139 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 <"Purity/0FRt2SRqqt0FBt/event_purity‚0.0032648316118866205 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.01778111420571804 9 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.054712992161512375 +jet/accuracy_2_of_3‚0.11163829267024994 jet/accuracy_4_of_4‚0 7validation_average_jet_accuracy‚0.10199113935232162 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9918867945671082 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.017089461907744408 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.05348251014947891 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.20706531405448914 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.014996564015746117 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_3‚1 $ _timestamp‚1.784330541057688e+09 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.11021973192691803 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.1825774908065796 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.0811915248632431 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.016406986862421036 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_2_of_2‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0260416679084301 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6719073057174683 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13919182121753693 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 +jet/accuracy_2_of_4‚0.10876293480396271 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9746651649475098 9 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.021666666492819786 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5318406224250793 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.03125 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10185423493385315 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14507147669792175 epoch‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9942382574081421 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.16511715948581696 ;"Purity/*FRt1SRqqt*FBt/event_purity‚0.041634876281023026 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8933023810386658 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.012453458271920681 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.0357142873108387 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.06548190861940384 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.09062342345714569 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9914021492004395 jet/accuracy_1_of_5‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.05178303271532059 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/sensitivity‚0.5475980043411255 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.976111114025116 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.01927661895751953 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0006265664123930037 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.01739204116165638 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.07495222985744476 +jet/accuracy_2_of_2‚0.04151742905378342 trainer/global_step‚399 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.28162339329719543 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9888668060302734 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9907406568527222 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13593067228794098 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.05573638901114464 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.28240740299224854 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0013182746479287744 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,jet/accuracy_3_of_3‚0.003582273842766881 particle/accuracy_3_of_3‚1 jet/accuracy_3_of_5‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.05217580497264862 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.013980885036289692 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_2‚1 particle/accuracy_3_of_4‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.16511717438697815 ("Purity/1FRt1SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +jet/accuracy_1_of_1‚0.07375412434339523 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9769269824028015 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.12862715125083923 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9896469116210938 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.1485089212656021 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0011614401591941714 (particle/f_score‚0.44964271783828735 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.09375 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14150716364383698 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 *loss/val_total_loss‚35.267917333397996 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10031089186668396 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9924991130828857 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1467539221048355 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06723356246948242 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.09018167108297348 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.08323611319065094 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.09330090880393982 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.17841175198554993 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9872353672981262 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.0786663293838501 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.4824557602405548 _runtime‚ 701.643817739 ("Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9876728057861328 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.10939793288707733 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.07266980409622192 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.06925841420888901iX—1ý™z 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.01778111420571804 9 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.054712992161512375 +jet/accuracy_2_of_3‚0.11163829267024994 jet/accuracy_4_of_4‚0 7validation_average_jet_accuracy‚0.10199113935232162 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9918867945671082 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.017089461907744408 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.05348251014947891 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.20706531405448914 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.014996564015746117 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_3‚1 $ _timestamp‚1.784330541057688e+09 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.11021973192691803 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.1825774908065796 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.0811915248632431 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.016406986862421036 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_2_of_2‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0260416679084301 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6719073057174683 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13919182121753693 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 +jet/accuracy_2_of_4‚0.10876293480396271 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9746651649475098 9 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.021666666492819786 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5318406224250793 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.03125 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10185423493385315 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14507147669792175 epoch‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9942382574081421 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.16511715948581696 ;"Purity/*FRt1SRqqt*FBt/event_purity‚0.041634876281023026 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8933023810386658 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.012453458271920681 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.0357142873108387 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.06548190861940384 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.09062342345714569 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9914021492004395 jet/accuracy_1_of_5‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.05178303271532059 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/sensitivity‚0.5475980043411255 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.976111114025116 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.01927661895751953 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0006265664123930037 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.01739204116165638 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.07495222985744476 +jet/accuracy_2_of_2‚0.04151742905378342 trainer/global_step‚399 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.28162339329719543 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9888668060302734 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9907406568527222 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13593067228794098 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.05573638901114464 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.28240740299224854 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0013182746479287744 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,jet/accuracy_3_of_3‚0.003582273842766881 particle/accuracy_3_of_3‚1 jet/accuracy_3_of_5‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.05217580497264862 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.013980885036289692 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_2‚1 particle/accuracy_3_of_4‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.16511717438697815 ("Purity/1FRt1SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +jet/accuracy_1_of_1‚0.07375412434339523 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9769269824028015 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.12862715125083923 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9896469116210938 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.1485089212656021 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0011614401591941714 (particle/f_score‚0.44964271783828735 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.09375 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14150716364383698 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 *loss/val_total_loss‚35.267917333397996 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10031089186668396 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9924991130828857 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1467539221048355 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06723356246948242 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.09018167108297348 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.08323611319065094 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.09330090880393982 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.17841175198554993 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9872353672981262 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.0786663293838501 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.4824557602405548 _runtime‚ 701.643817739 ("Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9876728057861328 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.10939793288707733 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.07266980409622192 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.06925841420888901 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9493302702903748 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9968253374099731 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.08131343126296997 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.016203705221414566 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.10745243728160858 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9767648577690125 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9931907653808594 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.03956804424524307 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_2_of_4‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.07706161588430405 ;"Purity/1FRt*SRqqt1FBt/event_purity‚0.010296827182173729 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.14944547414779663 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_3_of_5‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9900862574577332 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.07653003931045532 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.14351852238178253 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.07120181620121002 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.016841910779476166 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.0624452643096447 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9841940999031067 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.094841368496418 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11045478284358978 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0011614401591941714 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08270248025655746 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.06925841420888901 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.2770010232925415 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.10406067222356796 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9963717460632324 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0012954147532582283 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7477262616157532 <"Purity/0FRt2SRqqt*FBt/event_purity‚0.0031704306602478027 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.0748816579580307 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9943122863769531 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.1536659151315689 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08229292929172516 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7181670665740967 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.1917344331741333 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.20000000298023224 +jet/accuracy_1_of_2‚0.29250505566596985 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7486076354980469 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.012626002542674541 +particle/specificity‚0.7571113109588623 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.07176071405410767 ="Purity/1FRt1SRqqt0FBt/event_purity‚0.00046650946023873985 ) Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 _step‚32 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.4901600480079651 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purityƒ­Ì /*FRt*SRqqt0FBt/SRqqt_purity‚0.08842748403549194 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0006349206669256091 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_3_of_4‚0.002801120514050126 /particle/accuracy_4_of_4‚0.9893046021461487 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.2167254388332367 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.08706612139940262 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.12081970274448395 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9867726564407349 ="Purity/1FRt1SRqqt*FBt/event_purity‚0.00044689406058751047 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.01892898604273796 ;"Purity/*FRt*SRqqt0FBt/event_purity‚0.025126900523900986 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.1689140498638153 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11234768480062485 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.10156447440385818 particle/accuracy_4_of_5‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10986337065696716 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.28240740299224854 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08270248025655746 (particle/accuracy‚0.7121155858039856 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9963901042938232 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.12857992947101593 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9899304509162903 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.017089461907744408 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.12081970274448395 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.05337870866060257 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.10821931809186935 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.09116145223379135 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9924991130828857 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.10169368982315063 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.993740975856781 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.30000001192092896 ) Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.09263042360544205 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.08086773008108139 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 <"Purity/0FRt2SRqqt0FBt/event_purity‚0.0032648316118866205 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >U‘þ‚(:† °òêÒ‹—ÀÈ gpu.0.gpu‚7.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚30.0 gpu.0.powerWatts‚91.796' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3592 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UÍáÿ‚(:Ö °òêÒñýÀÊ disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 958.05859375 network.sent‚648396 disk.rbd2.out‚0 disk.drbd2165.out‚0! disk.dm-0.out‚357.30029296875& proc.memory.availableMB‚ 1674.46875 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.sda1.in‚0$ memory_percent‚89.77985382080078* proc.memory.percent‚18.729710578918457 cpu‚25.511684889700597$ disk./.usageGB‚581.8994255065918 disk.dm-1.out‚ 15.76953125 disk.sda3.out‚ 15.76953125# proc.memory.rssMB‚ 3068.67578125 disk.rbd1.out‚1.90625& disk.nvme0c0n1.out‚357.30029296875) disk./.usagePercent‚39.05175869188588 disk.sda.in‚0 disk.nvme0n1.in‚ 958.05859375 disk.sda3.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 958.05859375 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 15.76953125$ disk.nvme0n1.out‚357.30029296875 network.recv‚224546Pµ¿˜”€‚(:‰ ¿òêÒ¹Ê¶È gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚270.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿfòß‚(:Ô ¿òêÒƒóÊÉ network.sent‚705528 disk.dm-1.in‚0 disk.rbd1.in‚0$ disk./.usageGB‚582.8413887023926 disk.dm-1.out‚ 16.1796875# proc.memory.rssMB‚ 2798.61328125 disk.sda3.in‚0 disk.rbd2.out‚0# memory_percent‚99.9507188796997 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2165.in‚0! disk.dm-0.out‚2361.0107421875 disk.sda.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚17.081379890441895 disk.rbd1.out‚2.078125& disk.nvme0c0n1.out‚2361.0107421875* disk./.usagePercent‚39.114974632315835 disk.nvme0n1.in‚ 1000.0859375$ disk.nvme0n1.out‚2361.0107421875 disk.sda1.in‚0 disk.sda3.out‚ 16.1796875 disk.rbd0.in‚0 network.recv‚228963 disk.rbd2.in‚0& proc.memory.availableMB‚ 8.07421875 cpu‚25.508496272195806 disk.sda2.out‚0 disk.sda.out‚ 16.1796875" disk.nvme0c0n1.in‚ 1000.0859375 disk.dm-0.in‚ 1000.0859375 proc.cpu.threads‚17$õr‡‚‚(:’ ÎòêÒ»šÙÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚301.52' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.30399999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØÝAÚƒ‚(:Ï ÎòêÒ®Ø¯Ê disk.sda3.in‚0 disk.sda3.out‚ 16.2890625 disk.sda2.out‚0 disk.dm-0.in‚ 1096.71484375 disk.sda1.out‚0# memory_percent‚99.9521255493164 proc.cpu.threads‚17* disk./.usagePercent‚39.115039658229534 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚0 cpu‚25.50597148668084# disk.nvme0c0n1.in‚ 1096.71484375 disk.sda2.in‚0 disk.dm-1.in‚0* proc.memory.percent‚17.081379890441895 network.recv‚232235 disk.drbd2165.in‚0 disk.dm-0.out‚2365.474609375# proc.memory.rssMB‚ 2798.61328125 network.sent‚710595 disk.rbd1.out‚2.078125 disk.rbd2.in‚0 disk.dm-1.out‚ 16.2890625 disk.sda.in‚0 disk.sda.out‚ 16.2890625# proc.memory.availableMB‚7.84375 disk.rbd1.in‚0# disk./.usageGB‚582.842357635498% disk.nvme0c0n1.out‚2365.474609375! disk.nvme0n1.in‚ 1096.71484375# disk.nvme0n1.out‚2365.474609375 disk.sda1.in‚0.²å„‚(:Ú ÝòêÒúæØÉ disk.sda3.out‚ 16.58984375 disk.rbd2.out‚0 network.recv‚235669 disk.dm-0.in‚ 1097.6796875 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚2.140625$ memory_percent‚99.95272159576416& proc.memory.availableMB‚ 7.74609375# proc.memory.rssMB‚ 2798.61328125 network.sent‚715563 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0 cpu‚25.50332269579895 disk.rbd0.out‚0 disk.dm-1.out‚ 16.58984375 disk.sda.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.rbd2.in‚0 disk.sda.out‚ 16.58984375* proc.memory.percent‚17.081379890441895 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 1097.6796875* disk./.usagePercent‚39.115039402221996' disk.nvme0c0n1.out‚2374.09912109375$ disk./.usageGB‚582.8423538208008" disk.dm-0.out‚2374.09912109375 disk.nvme0n1.in‚ 1097.6796875% disk.nvme0n1.out‚2374.09912109375 disk.sda3.in‚0©‚äüž…‚(:“ ÝòêÒ–±øÊ gpu.0.gpu‚97.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚283.847' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.76939999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>³¾ì”†‚(:‰ ìòêÒÚÊÇÇ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~’và‡‚(:Õ ìòêÒ»¶ŸÊ' disk.nvme0c0n1.out‚2379.93994140625 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 1097.6796875 network.sent‚720576" disk.dm-0.out‚2379.93994140625 disk.nvme0n1.in‚ 1097.6796875 disk.sda3.in‚0 disk.rbd2.out‚0 disk.sda2.out‚0 cpu‚25.500691568841454 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 16.8828125# memory_percent‚99.9514102935791 disk.sda3.out‚ 16.8828125% proc.memory.availableMB‚ 7.9609375$ disk./.usageGB‚582.8424415588379 disk.rbd1.out‚2.140625) disk./.usagePercent‚39.11504529039529 disk.sda1.out‚0 disk.dm-1.out‚ 16.8828125# proc.memory.rssMB‚ 2798.61328125* proc.memory.percent‚17.081379890441895 network.recv‚238893% disk.nvme0n1.out‚2379.93994140625 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 1097.6796875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0yoÄøŸˆ‚(:” ûòêÒ‰ïºÈ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.567' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.113400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QÔ‰‚(:É ûòêÒàšÊÉ proc.cpu.threads‚17 network.recv‚242542 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.sda1.in‚0) disk./.usagePercent‚39.11505169058364 disk.rbd1.out‚2.21875# disk.nvme0n1.out‚2387.853515625 disk.sda3.in‚0 disk.sda3.out‚ 17.1015625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2798.61328125* proc.memory.percent‚17.081379890441895 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.dm-0.out‚2387.853515625 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.nvme0c0n1.in‚ 1098.71875% disk.nvme0c0n1.out‚2387.853515625% proc.memory.availableMB‚ 7.9453125 cpu‚25.498238583051627 disk.rbd2.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 1098.71875 disk.rbd0.in‚0 network.sent‚725564$ disk./.usageGB‚582.8425369262695 disk.dm-1.out‚ 17.1015625 disk.sda.out‚ 17.1015625 disk.nvme0n1.in‚ 1098.71875$ memory_percent‚99.95150566101074G¶Î0Š+ _wandbruntime‚788  _runtime‚788'ZšÅ‹¿ ' lr-AdamW/pg1‚0.00048246595594668906 ' lr-AdamW/pg2‚0.00048246595594668906 trainer/global_step‚449 $ _timestamp‚1.784330628027305e+09 _runtime‚ 788.612453731 _step‚33e>Ɍà trainer/global_step‚449 $ _timestamp‚1.784330628027305e+09 _runtime‚ 788.612453731 _step‚33 ' lr-AdamW/pg1‚0.00048246595594668906 ' lr-AdamW/pg2‚0.00048246595594668906!=€ÐÒ0+ _wandbruntime‚788  _runtime‚788/‘þnÒ'ŽÌO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129380455444685 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129437767161856 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7716103 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7726080 _runtime‚ 788.616928325 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3897003 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2437771 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35109115154432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34537472 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129015812227072 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 421981108224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3829071 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚449 % _timestamp‚1.7843306280286071e+09 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1157472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128959197515776 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3897003 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129437767161856 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129015786053632 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129015786053632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35108747063296 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129439138723840 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2438024 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 542971573760 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚35651718637056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚129381825472967 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3829077 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1280552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1157467 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _step‚34 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 421981108224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129439138723840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129015812227072 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0]Ô!Ö'ÐO % _timestamp‚1.7843306280286071e+09 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1157472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128959197515776 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3897003 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129437767161856 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129015786053632 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129015786053632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35108747063296 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129439138723840 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2438024 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 542971573760 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚35651718637056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚129381825472967 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3829077 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1280552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1157467 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _step‚34 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 421981108224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129439138723840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129015812227072 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129380455444685 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129437767161856 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7716103 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7726080 _runtime‚ 788.616928325 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3897003 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2437771 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35109115154432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34537472 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129015812227072 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 421981108224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3829071 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚449"‚Á5b0+ _wandbruntime‚790  _runtime‚790)\òÅ‘ÖM F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7742778 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 422181395155 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 422906143744 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129660630717501 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1160010 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2443071 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35186233470976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129716692927488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35730408394240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 544174923264 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1160015 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 544209453568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129293786783744 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3895510 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚ 790.047063145 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3837277 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3905495 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 422906143744 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3837283 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35186601562112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 424251539456 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7732787 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3837277 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35730811015680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402621440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129293812957184 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3905495 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚449 % _timestamp‚1.7843306294617324e+09 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7742778 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129718064496640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129718064496640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34530304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7732787 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3895510 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129293786783744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3837283 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129659260689107 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129237079293952 _step‚35 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 422181395155 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 422906143744 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129660630717501 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1160010 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2443071 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35186233470976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129716692927488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35730408394240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 544174923264 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1160015 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 544209453568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129293786783744 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3895510 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚ 790.047063145 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3837277 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3905495 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 422906143744 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3837283 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35186601562112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 424251539456 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7732787 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3837277 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35730811015680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402621440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129293812957184 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3905495 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚449 % _timestamp‚1.7843306294617324e+09 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7742778 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129718064496640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129718064496640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34530304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7732787 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3895510 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129293786783744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3837283 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129659260689107 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129237079293952 _step‚35 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 Et.›ß¡‚(:Ô ÕóêÒœìéÉ disk.dm-1.out‚ 18.703125" proc.memory.rssMB‚ 2798.6171875 proc.cpu.threads‚17 network.sent‚813377 disk.rbd0.out‚0 disk.rbd1.out‚2.421875" disk.dm-0.out‚2433.11572265625* proc.memory.percent‚17.081403732299805 disk.dm-1.in‚0 disk.dm-0.in‚ 1305.48828125 disk.rbd0.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1305.48828125! disk.nvme0n1.in‚ 1305.48828125$ proc.memory.availableMB‚6.421875$ disk./.usageGB‚582.8429107666016 disk.sda1.in‚0 disk.rbd2.in‚0 disk.drbd2165.in‚0* disk./.usagePercent‚39.115076779321996 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 18.703125' disk.nvme0c0n1.out‚2433.11572265625 network.recv‚263982 disk.sda.in‚0% disk.nvme0n1.out‚2433.11572265625 disk.sda3.out‚ 18.703125# memory_percent‚99.9608039855957 cpu‚25.484996155453906 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd2.out‚0C2~³0¢+ _wandbruntime‚871  _runtime‚871Üÿy6Å£¿ ' lr-AdamW/pg2‚0.00048244302503315257 trainer/global_step‚499 % _timestamp‚1.7843307110744524e+09 _runtime‚ 871.65962966 _step‚37 ' lr-AdamW/pg1‚0.00048244302503315257’¨Ïɤà ' lr-AdamW/pg2‚0.00048244302503315257 trainer/global_step‚499 % _timestamp‚1.7843307110744524e+09 _runtime‚ 871.65962966 _step‚37 ' lr-AdamW/pg1‚0.00048244302503315257%9`v0¥+ _wandbruntime‚871  _runtime‚871ñz ¦ËO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4322755 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143320781466593 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚142915848732672 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4239371 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4239371 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1419453 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142915822559232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚142915848732672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 469642427392 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 468297038848 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 467495042017 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2704351 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4239377 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39586034168832 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 $ _timestamp‚1.784330711076292e+09 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8552155 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4322755 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143384119598080 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚143322151494875 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 469642427392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1419729 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚ 871.661763788 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142915822559232 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8562132 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4312784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚143385491160064 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4312784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚38 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 602966721536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4239377 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 [Õ'§ÏO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143320781466593 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚142915848732672 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4239371 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4239371 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1419453 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142915822559232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚142915848732672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 469642427392 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 468297038848 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 467495042017 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2704351 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4239377 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39586034168832 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 $ _timestamp‚1.784330711076292e+09 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8552155 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4322755 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143384119598080 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚143322151494875 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 469642427392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1419729 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚ 871.661763788 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142915822559232 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8562132 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4312784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚143385491160064 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4312784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚38 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 602966721536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4239377 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4322755 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0&þ0¨+ _wandbruntime‚873  _runtime‚873í %ÒÜ&©ÖM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39060553854976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 470567470080 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 469763108689 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143599586711015 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4247577 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚143664416932864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 468418508263 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚39 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1287165 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143131193630720 trainer/global_step‚499 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143193823289344 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8578830 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2709352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1287160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1422458 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4247583 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4247577 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143600956739409 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8568839 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2709623 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143131168202752 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 469222074368 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 469222074368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚ 873.062938846 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚143663045363712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 604195515392 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39665117461504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1422192 % _timestamp‚1.7843307124775288e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚143664416932864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 468418508263 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚39 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1287165 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143131193630720 trainer/global_step‚499 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143193823289344 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8578830 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2709352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1287160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1422458 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4247583 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4247577 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143600956739409 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8568839 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2709623 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143131168202752 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 469222074368 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 469222074368 @:DeviceStatsMonitor.on_train_ba3˜7O;tch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚ 873.062938846 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚143663045363712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 604195515392 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39665117461504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1422192 % _timestamp‚1.7843307124775288e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 470567470080 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 469763108689 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143599586711015 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995'Q —½0«+ _wandbruntime‚873  _runtime‚8732›µ“¬ú 1loss/SRqqt1/detection_loss‚2.6382703709650817 1loss/SRqqt2/detection_loss‚0.7614834356044289 epoch‚1 trainer/global_step‚499 /loss/FRt1/assignment_loss‚3.986679203586828 /loss/FRt2/assignment_loss‚3.037472877800943 0loss/FBt2/assignment_loss‚0.2835864226333844 /loss/FRt2/detection_loss‚1.4164914455418782 %loss/total_loss‚21.15032839920614 2loss/SRqqt2/assignment_loss‚1.5243497362245966 /loss/FRt1/detection_loss‚0.6338096450593191 .loss/FBt2/detection_loss‚2.673567236346357 2loss/SRqqt1/assignment_loss‚1.8966050104397099 1loss/FBt1/assignment_loss‚0.06876145802161494 /loss/FBt1/detection_loss‚2.2292515569819944 % _timestamp‚1.7843307125603714e+09 _runtime‚ 873.145436726 _step‚40N”ŒG­þ /loss/FRt1/assignment_loss‚3.986679203586828 /loss/FRt2/assignment_loss‚3.037472877800943 0loss/FBt2/assignment_loss‚0.2835864226333844 /loss/FRt2/detection_loss‚1.4164914455418782 %loss/total_loss‚21.15032839920614 2loss/SRqqt2/assignment_loss‚1.5243497362245966 /loss/FRt1/detection_loss‚0.6338096450593191 .loss/FBt2/detection_loss‚2.673567236346357 2loss/SRqqt1/assignment_loss‚1.8966050104397099 1loss/FBt1/assignment_loss‚0.06876145802161494 /loss/FBt1/detection_loss‚2.2292515569819944 % _timestamp‚1.7843307125603714e+09 _runtime‚ 873.145436726 _step‚40 1loss/SRqqt1/detection_loss‚2.6382703709650817 1loss/SRqqt2/detection_loss‚0.7614834356044289 epoch‚1 trainer/global_step‚499(ñÙŸ®‚(:” äóêÒù×ñÇ gpu.0.gpu‚73.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚111.353' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.270599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«%Z4د‚(:Í äóêÒèÇ«É) disk./.usagePercent‚39.11508292350282 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.drbd2165.out‚0 disk.dm-0.out‚2443.095703125# proc.memory.availableMB‚6.59375 proc.cpu.threads‚17 network.sent‚817318 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 1306.453125* proc.memory.percent‚17.081403732299805 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda.out‚ 18.98046875$ memory_percent‚99.95975494384766 cpu‚25.483074697991693$ disk./.usageGB‚582.8430023193359 disk.dm-1.out‚ 18.98046875% disk.nvme0c0n1.out‚2443.095703125 disk.nvme0n1.in‚ 1306.453125 disk.sda3.in‚0" proc.memory.rssMB‚ 2798.6171875# disk.nvme0n1.out‚2443.095703125 disk.rbd2.out‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 1306.453125 network.recv‚266450 disk.sda3.out‚ 18.98046875 disk.rbd1.out‚2.421875¨‚¤8ž°‚(:“ óóêÒéöŒÇ gpu.0.gpu‚70.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚330.288' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.05760000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@ãŽØܱ‚(:Ñ óóêÒý‘úÉ network.sent‚880083$ disk./.usageGB‚582.8434982299805 disk.rbd2.out‚0 disk.sda2.out‚0 disk.drbd2165.in‚0" disk.dm-0.out‚2447.67919921875 disk.sda3.in‚0 disk.sda3.out‚ 19.26171875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚2447.67919921875 disk.dm-1.out‚ 19.26171875 disk.sda.in‚0 disk.dm-0.in‚ 1306.453125 cpu‚25.4819637025392 proc.cpu.threads‚17) disk./.usagePercent‚39.11511620448227 disk.nvme0n1.in‚ 1306.453125$ memory_percent‚99.95980262756348 disk.sda1.out‚0 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 1306.453125 network.recv‚270992 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚2.5625 disk.sda.out‚ 19.26171875% proc.memory.availableMB‚ 6.5859375" proc.memory.rssMB‚ 2798.6171875% disk.nvme0n1.out‚2447.67919921875* proc.memory.percent‚17.081403732299805@ºs…”²‚(:‰ ‚ôêÒß®ÃÇ gpu.0.gpu‚88.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚322.472' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.4944 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿt°Û³‚(:Ð ‚ôêÒ׺—Ê disk.sda3.out‚ 19.5546875 disk.rbd2.out‚0* proc.memory.percent‚17.081403732299805 proc.cpu.threads‚17 disk.sda3.in‚0 disk.dm-1.out‚ 19.5546875* disk./.usagePercent‚39.115090347721306 disk.sda.in‚0 disk.dm-0.out‚2459.236328125 disk.rbd1.in‚0 disk.sda.out‚ 19.5546875& proc.memory.availableMB‚ 6.57421875" proc.memory.rssMB‚ 2798.6171875# disk.nvme0c0n1.in‚ 1371.95703125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd1.out‚2.5625 disk.drbd2165.in‚0# disk.nvme0n1.out‚2459.236328125 disk.sda1.out‚0 disk.rbd0.out‚0 network.recv‚273949 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 cpu‚25.48011019396454 network.sent‚885059! disk.nvme0n1.in‚ 1371.95703125 disk.drbd2165.out‚0# memory_percent‚99.9598741531372% disk.nvme0c0n1.out‚2459.236328125 disk.dm-0.in‚ 1371.95703125$ disk./.usageGB‚582.8431129455566&I¿ž´‚(:“ ‘ôêÒ³‰·Ç gpu.0.gpu‚84.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚317.192' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚63.43840000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß0ûݵ‚(:Ò ‘ôêÒ¿ƒñÉ disk.rbd2.in‚0 disk.nvme0n1.in‚ 1372.921875 disk.sda1.out‚0 disk.dm-1.out‚ 19.86328125 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚2468.30517578125 disk.sda.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚582.8432273864746 disk.sda3.in‚0 cpu‚25.478345651962766 disk.rbd2.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 1372.921875% disk.nvme0n1.out‚2468.30517578125 disk.sda2.out‚0 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 1372.921875" proc.memory.rssMB‚ 2798.6171875 network.sent‚890177) disk./.usagePercent‚39.11509802794734 disk.sda1.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.081403732299805 network.recv‚277260 disk.sda3.out‚ 19.86328125 disk.rbd0.in‚0 disk.sda.out‚ 19.86328125 disk.drbd2165.out‚0" disk.dm-0.out‚2468.30517578125$ memory_percent‚99.95713233947754 disk.rbd0.out‚0 disk.rbd1.out‚2.625% proc.memory.availableMB‚ 7.0234375©µÚž¶‚(:“  ôêÒæ²õÈ gpu.0.gpu‚95.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚322.848' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚64.56960000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J*]Ò·‚(:Ç  ôêÒÕðÉ disk.rbd1.out‚2.625 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda3.in‚0$ memory_percent‚99.95176792144775& proc.memory.availableMB‚ 7.90234375# proc.memory.rssMB‚ 2798.62109375 network.recv‚280541! disk.nvme0c0n1.in‚ 1372.921875 disk.sda3.out‚20.0625 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 1372.921875 disk.rbd0.in‚0 cpu‚25.47687675761018 network.sent‚895250 disk.rbd1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚39.11508343551789 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 1372.921875$ disk./.usageGB‚582.8430099487305 disk.sda.in‚0 disk.sda.out‚20.0625' disk.nvme0c0n1.out‚2474.92822265625 disk.rbd0.out‚0 disk.rbd2.in‚0" disk.dm-0.out‚2474.92822265625 disk.rbd2.out‚0 disk.sda2.out‚0% disk.nvme0n1.out‚2474.92822265625 disk.dm-1.out‚20.0625* proc.memory.percent‚17.081427574157715 proc.cpu.threads‚17jîf50¸+ _wandbruntime‚955  _runtime‚955ï™Ä¹¾ % _timestamp‚1.7843307944405847e+09 _runtime‚ 955.025833855 _step‚41 & lr-AdamW/pg1‚0.0004824046018879697 & lr-AdamW/pg2‚0.0004824046018879697 trainer/global_step‚549Kjx@Ⱥ trainer/global_step‚549 % _timestamp‚1.7843307944405847e+09 _runtime‚ 955.025833855 _step‚41 & lr-AdamW/pg1‚0.0004824046018879697 & lr-AdamW/pg2‚0.0004824046018879697)ÅóíŽ0»+ _wandbruntime‚955  _runtime‚955îc>-Ò'¼ÌO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9398184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚157262477516783 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4649677 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚42857387831296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156815859064832 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 514612969472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚549 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4738536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156815885238272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4738536 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 515076755439 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _runtime‚ 955.030039715 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157330472034304 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 513732155125 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚157331843596288 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 514612969472 % _timestamp‚1.7843307944420135e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156815885238272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157261107488501 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1411772 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4748507 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 662513772544 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4649671 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 515958358016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43520306329600 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1557713 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156815859064832 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1411767 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚156747375333376K¨“ü½ÐO OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4649677 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36634624 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚42857755922432 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 515958358016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚42857387831296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156815859064832 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 514612969472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚549 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4738536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156815885238272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4738536 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 515076755439 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _runtime‚ 955.030039715 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157330472034304 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 513732155125 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚157331843596288 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 514612969472 % _timestamp‚1.7843307944420135e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156815885238272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157261107488501 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1411772 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4748507 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 662513772544 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4649671 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 515958358016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43520306329600 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1557713 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156815859064832 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1411767 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚156747375333376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9398184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚157262477516783 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4649677 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚157610769369088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 trainer/global_step‚549 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4747014 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 516883400704 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚157093859794944 $ _timestamp‚1.784330795879709e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4747014 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2974735 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 663700940800 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4657883 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚157025257111552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9404891 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9414882 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚42934874238976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2975005 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157539912732923 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43598979898368 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43598575179776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚157093885968384«€ý[ß&ÀÙM MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4756999 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4657877 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚157610769369088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 trainer/global_step‚549 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4747014 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 516883400704 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚157093859794944 $ _timestamp‚1.784330795879709e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4747014 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2974735 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 663700940800 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4657883 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚157025257111552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9404891 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9414882 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚42934874238976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2975005 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157539912732923 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43598979898368 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43598575179776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚157093885968384 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1414310 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 516000221797 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _runtime‚ 956.465084776 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 559969268233 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 721945773056 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1538922 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1538917 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5059977 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10234236 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 561313868547 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5059977 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5164288 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 562274288640 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚170641464242176 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5059971 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36634624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1695434 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚47453617353728 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 trainer/global_step‚599 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 1038.11747207 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚171276824470528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46731708215296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171202803538691 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5174259 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3234351 _step‚46 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170715895570432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170715921743872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1695708 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 721909138432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171201433510409 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 560928900096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚47454022079488 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5174259àG½Ù'ÕÓO RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 560928900096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚47454022079488 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5174259 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46732076306432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 559969268233 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 721945773056 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1538922 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1538917 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5059977 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10234236 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 561313868547 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5059977 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5164288 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 562274288640 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚170641464242176 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5059971 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36634624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1695434 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚47453617353728 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 trainer/global_step‚599 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 1038.11747207 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚171276824470528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46731708215296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171202803538691 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5174259 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3234351 _step‚46 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170715895570432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170715921743872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1695708 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 721909138432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171201433510409 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175.?I8?2Ö- _wandbruntime‚1039  _runtime‚1039;òÜÄá&×ÛM WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 561853935616 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5172766 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _step‚47 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170993896300544 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3239642 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 723091105792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚47532690447360 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 723127733248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚1039.522416205 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170993896300544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 563199331328 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1541460 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170993922473984 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 563199331328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170919371448320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5068177 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚599 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10240943 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10250934 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1541465 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 562237334905 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170993922473984 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1698441 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 % _timestamp‚1.7843308789370613e+09 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5182751 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5172766 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3239906 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10240943 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚47532285728768 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5068183 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5068177 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚170919346020352 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46809562714112 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10250934!KÝþ¾ ØßM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚171557121805312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚171481608783225 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚171555750236160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171480238754831 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 560892734479 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5068183 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.currôÊ`á'ent‚ 1345395712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5068177 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚170919346020352 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46809562714112 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10250934 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 561853935616 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5172766 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _step‚47 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170993896300544 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3239642 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 723091105792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚47532690447360 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 723127733248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚1039.522416205 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170993896300544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 563199331328 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1541460 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170993922473984 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 563199331328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170919371448320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5068177 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚599 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10240943 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10250934 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1541465 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 562237334905 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170993922473984 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1698441 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 % _timestamp‚1.7843308789370613e+09 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5182751 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5172766 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3239906 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10240943 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚47532285728768 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 781227092480 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 781190457856 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 607550981655 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185223176906752 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185223176906752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 % _timestamp‚1.7843309606050804e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 607244830720 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3498791 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1666067 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5470277 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚185141759532317 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚184535578578944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚185143129560599 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 trainer/global_step‚649 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184615958249472 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1832988 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50606028599296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36634624 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1832724 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51387219057152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5470271 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5600011 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184615958249472 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51387623782912 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _step‚50 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184615932076032 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185224548468736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184615932076032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 781227092480 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 781190457856 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 607550981655 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185223176906752 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185223176906752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 % _timestamp‚1.7843309606050804e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 607244830720 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3498791 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1666067 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5470277 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UL[ç"ßADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚185141759532317 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚184535578578944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚185143129560599 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 trainer/global_step‚649 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184615958249472 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1832988 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50606028599296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36634624 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1832724 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51387219057152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5470271 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5600011 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184615958249472 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51387623782912 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _step‚50 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184615932076032 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185224548468736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184615932076032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚02Xà$2ì- _wandbruntime‚1122  _runtime‚1122’nG=á&íÛM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚184813460357120 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1668615 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 782409773568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚51466292871680 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 608474448013 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5608503 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185421934805133 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11086986 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7843309620051093e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185502102672384 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3504074 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5598518 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1835719 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185420564776739 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _step‚51 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3504334 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 607129847587 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5478483 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚184893958979584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚184813434929152 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185503474241536 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5608503 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚184893932806144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50683883098112 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50683515006976 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185503474241536 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 609515261952 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11076995 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚184893958979584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5598518 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5478477 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1668610 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51465888153088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0K¨¾qå&îßM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚51466292871680 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 608474448013 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5608503 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185421934805133 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11086986 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7843309620051093e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185502102672384 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3504074 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5598518 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1835719 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185420564776739 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _step‚51 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3504334 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 607129847587 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5478483 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚184893958979584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚184813434929152 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185503474241536 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5608503 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚184893932806144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50683883098112 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50683515006976 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185503474241536 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 609515261952 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11076995 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚184893958979584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5598518 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5478477 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1668610 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51465888153088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚184813460357120 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1668615 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 782409773568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚03,—½V2ï- _wandbruntime‚1122  _runtime‚1122KúÙßðþ 0loss/FRt2/assignment_loss‚2.7005517026661217 1loss/FBt2/assignment_loss‚0.19531511296478332 &loss/total_loss‚18.568457678917852 trainer/global_step‚649 0loss/FRt1/assignment_loss‚3.6662037348691023 /loss/FRt2/detection_loss‚1.4365935304809387 1loss/SRqqt2/detection_loss‚0.6900077533618302 /loss/FBt1/detection_loss‚1.0209041092983466 epoch‚1 2loss/SRqqt1/assignment_loss‚1.7354486619302056 2loss/SRqqt2/assignment_loss‚1.4882453278626497 1loss/FBt1/assignment_loss‚0.11584159289224424 /loss/FRt1/detection_loss‚0.6540763431270157 .loss/FBt2/detection_loss‚2.606981622179349 % _timestamp‚1.7843309620871732e+09 _runtime‚ 1122.67226645 _step‚52 1loss/SRqqt1/detection_loss‚2.2582881872852645b¾]ˆñ‚ 0loss/FRt2/assignment_loss‚2.7005517026661217 1loss/FBt2/assignment_loss‚0.19531511296478332 &loss/total_loss‚18.568457678917852 trainer/global_step‚649 0loss/FRt1/assignment_loss‚3.6662037348691023 /loss/FRt2/detection_loss‚1.4365935304809387 1loss/SRqqt2/detection_loss‚0.6900077533618302 /loss/FBt1/detection_loss‚1.0209041092983466 epoch‚1 2loss/SRqqt1/assignment_loss‚1.7354486619302056 2loss/SRqqt2/assignment_loss‚1.4882453278626497 1loss/FBt1/assignment_loss‚0.11584159289224424 /loss/FRt1/detection_loss‚0.6540763431270157 .loss/FBt2/detection_loss‚2.606981622179349 % _timestamp‚1.7843309620871732e+09 _runtime‚ 1122.67226645 _step‚52 1loss/SRqqt1/detection_loss‚2.25828818728526454yÌr¿”ò‚(:‰ ÔõêÒÙÎüÊ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚339.562' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.9124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷Íf·ëó‚(:à ÔõêÒŸ €Ì* proc.memory.percent‚17.081475257873535 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚0 disk.rbd1.out‚3.109375& disk.nvme0c0n1.out‚3406.9150390625$ disk.nvme0n1.out‚3406.9150390625 disk.sda1.out‚0$ memory_percent‚99.95853900909424* disk./.usagePercent‚39.116849631496606 disk.sda.out‚ 27.04296875 disk.sda3.in‚0 network.recv‚322566 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 27.04296875 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 1583.90234375 cpu‚25.461488610726605 proc.cpu.threads‚17# disk./.usageGB‚582.869327545166 disk.drbd2165.out‚0 disk.sda3.out‚ 27.04296875 disk.dm-0.in‚ 1583.90234375! disk.dm-0.out‚3406.9150390625! disk.nvme0n1.in‚ 1583.90234375 disk.sda1.in‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 6.79296875# proc.memory.rssMB‚ 2798.62890625 network.sent‚ 1.083823e+06 disk.sda.in‚0 disk.rbd2.in‚0}Â4”ô‚(:‰ ãõêÒ™ü±Ç gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚331.111' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.2222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô à‘ãõ‚(:Ø ãõêÒêªçÉ disk.rbd2.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 27.4921875# disk.nvme0n1.out‚3410.080078125 disk.sda1.out‚0 network.sent‚ 1.133675e+06 disk.rbd1.in‚0 disk.dm-0.in‚ 1583.90234375 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 1583.90234375 disk.dm-0.out‚3410.080078125& proc.memory.availableMB‚ 6.49609375$ disk./.usageGB‚582.8692855834961 disk.sda1.in‚0 disk.sda3.out‚ 27.4921875 disk.rbd0.out‚0 disk.drbd2165.in‚0% disk.nvme0c0n1.out‚3410.080078125! disk.nvme0n1.in‚ 1583.90234375* proc.memory.percent‚17.081475257873535 proc.cpu.threads‚17) disk./.usagePercent‚39.11684681541374 disk.rbd0.in‚0 disk.sda.out‚ 27.4921875 disk.sda2.out‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2798.62890625 cpu‚25.460367068037428 network.recv‚327343 disk.rbd2.out‚0 disk.rbd1.out‚3.1875 disk.sda3.in‚0$ memory_percent‚99.96035099029541*ËÈú”ö‚(:‰ òõêÒËÀ¡Ç gpu.0.gpu‚84.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚341.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.2104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]¼ldÖ÷‚(:Ë òõêÒ칉É network.sent‚ 1.138699e+06$ disk./.usageGB‚582.8698043823242! disk.nvme0n1.in‚ 1675.98828125 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0* proc.memory.percent‚17.081475257873535 disk.dm-0.out‚ 3420.015625 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda.in‚0$ memory_percent‚99.95894432067871% proc.memory.availableMB‚ 6.7265625# proc.memory.rssMB‚ 2798.62890625 proc.cpu.threads‚17 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1675.98828125* disk./.usagePercent‚39.116881632438385 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.nvme0n1.out‚ 3420.015625 disk.dm-1.in‚0" disk.nvme0c0n1.out‚ 3420.015625 network.recv‚330697 disk.rbd2.out‚0 cpu‚25.45920933658525 disk.dm-0.in‚ 1675.98828125 disk.sda1.out‚0 disk.sda3.out‚ 27.765625 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 27.765625 disk.rbd1.out‚3.1875 disk.sda.out‚ 27.765625»»ýžø‚(:“ öêÒÖÍÙÇ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚285.199' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.03980000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓHNáù‚(:Ö öêÒèþ¾É% disk.nvme0n1.out‚3427.43896484375 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.dm-1.out‚ 28.19140625 disk.sda.out‚ 28.19140625 cpu‚25.45823843208847 disk.rbd1.out‚3.25 disk.drbd2165.out‚0* disk./.usagePercent‚39.116911329312366 disk.rbd2.out‚0 disk.sda2.out‚0 disk.drbd2165.in‚0 network.recv‚334187# disk./.usageGB‚582.870246887207 disk.rbd1.in‚0 disk.sda.in‚0! disk.nvme0c0n1.in‚ 1677.203125' disk.nvme0c0n1.out‚3427.43896484375 network.sent‚ 1.143674e+06" disk.dm-0.out‚3427.43896484375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 28.19140625 disk.dm-1.in‚0* proc.memory.percent‚17.081475257873535 disk.dm-0.in‚ 1677.203125 disk.nvme0n1.in‚ 1677.203125 disk.sda3.in‚0$ proc.memory.availableMB‚6.734375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda2.in‚0$ memory_percent‚99.95889663696289# proc.memory.rssMB‚ 2798.62890625kK÷FŸú‚(:” öêÒ¡ôäÇ gpu.0.gpu‚86.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚284.911' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.982200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o%üLöû‚(:Ñ öêÒÕ¡‚Ê disk.sda2.in‚0 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 1677.203125" disk.dm-0.out‚3473.49755859375 disk.sda1.in‚0 disk.rbd2.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda.out‚29.71875' disk.nvme0c0n1.out‚3473.49755859375 disk.sda3.in‚0$ memory_percent‚99.95553493499756# proc.memory.rssMB‚ 2798.62890625 network.sent‚ 1.148636e+06& proc.memory.availableMB‚ 7.28515625* proc.memory.percent‚17.081475257873535$ disk./.usageGB‚579.7539749145508 disk.sda2.out‚0 disk.sda3.out‚29.71875 disk.rbd0.in‚0 network.recv‚337178 disk.dm-1.out‚29.71875% disk.nvme0n1.out‚3473.49755859375 disk.rbd0.out‚0 disk.rbd2.out‚0 cpu‚25.457080zäûeæ301521017 proc.cpu.threads‚17* disk./.usagePercent‚38.907775702501056 disk.rbd1.out‚3.25 disk.sda.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 1677.203125 disk.nvme0n1.in‚ 1677.203125 disk.sda1.out‚0m|¾¯”ü‚(:‰ ŸöêÒš­ýÈ gpu.0.gpu‚86.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èØá©Öý‚(:Ë ŸöêÒûæÕÊ disk.dm-1.out‚ 31.04296875 disk.sda.in‚0 cpu‚25.456149078733354$ disk./.usageGB‚579.9620246887207 disk.dm-1.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2798.62890625* proc.memory.percent‚17.081475257873535" disk.nvme0c0n1.in‚ 1678.0546875 disk.nvme0c0n1.out‚ 3511.0625 disk.dm-0.in‚ 1678.0546875 disk.sda1.in‚0 network.recv‚340865 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 1678.0546875 network.sent‚ 1.153771e+06 disk.rbd1.in‚0 disk.dm-0.out‚ 3511.0625 disk.nvme0n1.out‚ 3511.0625 disk.sda3.in‚0 disk.sda3.out‚ 31.04296875) disk./.usagePercent‚38.92173809741099 disk.sda.out‚ 31.04296875 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0$ memory_percent‚99.95577335357666 disk.sda2.in‚0 disk.rbd1.out‚3.390625 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd2.out‚0& proc.memory.availableMB‚ 7.24609375D,ÖÁ2þ- _wandbruntime‚1204  _runtime‚1204/f oÇÿÁ _runtime‚1204.228271979 _step‚53 ' lr-AdamW/pg1‚0.00048219640871518155 ' lr-AdamW/pg2‚0.00048219640871518155 trainer/global_step‚699 % _timestamp‚1.7843310436429894e+09Àãü{ˀŠtrainer/global_step‚699 % _timestamp‚1.7843310436429894e+09 _runtime‚1204.228271979 _step‚53 ' lr-AdamW/pg1‚0.00048219640871518155 ' lr-AdamW/pg2‚0.000482196408715181555Œu7:2- _wandbruntime‚1204  _runtime‚1204˾Ü/×'‚ÑO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 654906149888 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚1204.230474969 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 654906149888 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5880577 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5880571 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6015792 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚199082085554225 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3762590 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1793222 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54480348983296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1793217 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚198429642059776 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199169529342976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198515968581632 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54480717074432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55321023255040 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11906340 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 % _timestamp‚1.7843310436447353e+09 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 653560761344 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6025763 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199169529342976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198515968581632 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 654906149888 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚1204.230474969 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 654906149888 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5880577 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5880571 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6015792 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚199082085554225 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3762590 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1793222 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54480348983296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1793217 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚198429642059776 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199169529342976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198515968581632 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54480717074432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55321023255040 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11906340 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 % _timestamp‚1.7843310436447353e+09 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 653560761344 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6025763 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199169529342976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 490712596486Àçü÷2„- _wandbruntime‚1205  _runtime‚1205’çg›á&…ÛM B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 trainer/global_step‚699 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11913047 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54558203482112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 _runtime‚1205.638068261 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198793995485184 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199449826677760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1972122 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11923038 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1795760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚198793995485184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11913047 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55400100060672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 841896578560 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚198793969311744 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7843310450526695e+09 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6024270 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11923038 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6024270 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 655831192576 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 654485796864 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚198707549265920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3768148 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3767882 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199448455108608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5888777 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5888783 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 654711561121 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 655831192576 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55399695342080 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008¦ 4¡†ßM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytesòºªûD&.small_pool.peak‚ 295881216 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5888783 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 653366960695 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚198793969311744 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7843310450526695e+09 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6024270 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11923038 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6024270 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 655831192576 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 654485796864 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚198707549265920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3768148 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3767882 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199448455108608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5888777 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5888783 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 654711561121 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 655831192576 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55399695342080 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 trainer/global_step‚699 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11913047 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54558203482112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 _runtime‚1205.638068261 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198793995485184 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199449826677760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1972122 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11923038 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1795760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚198793995485184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11913047 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55400100060672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 841896578560 BùQˆþ /loss/FRt2/detection_loss‚1.4881558278067273 1loss/SRqqt2/detection_loss‚0.6610815462331294 /loss/FBt1/detection_loss‚0.6898672272028792 /loss/FBt2/detection_loss‚1.9694454448838388 _step‚56 %loss/total_loss‚17.10549991173017 0loss/FRt1/assignment_loss‚3.6405492817350833 1loss/FBt1/assignment_loss‚0.17829925720730974 1loss/FBt2/assignment_loss‚0.15914961976933692 trainer/global_step‚699 % _timestamp‚1.7843310451335957e+09 _runtime‚1205.718690035 0loss/FRt2/assignment_loss‚2.5202784713351707 2loss/SRqqt1/assignment_loss‚1.6156880264696867 2loss/SRqqt2/assignment_loss‚1.3847433094110264 0loss/SRqqt1/detection_loss‚2.136873760430709 epoch‚1 /loss/FRt1/detection_loss‚0.6613681392452747°Þ|̉‚ /loss/FBt2/detection_loss‚1.9694454448838388 _step‚56 %loss/total_loss‚17.10549991173017 0loss/FRt1/assignment_loss‚3.6405492817350833 1loss/FBt1/assignment_loss‚0.17829925720730974 1loss/FBt2/assignment_loss‚0.15914961976933692 trainer/global_step‚699 % _timestamp‚1.7843310451335957e+09 _runtime‚1205.718690035 0loss/FRt2/assignment_loss‚2.5202784713351707 2loss/SRqqt1/assignment_loss‚1.6156880264696867 2loss/SRqqt2/assignment_loss‚1.3847433094110264 0loss/SRqqt1/detection_loss‚2.136873760430709 epoch‚1 /loss/FRt1/detection_loss‚0.6613681392452747 /loss/FRt2/detection_loss‚1.4881558278067273 1loss/SRqqt2/detection_loss‚0.6610815462331294 /loss/FBt1/detection_loss‚0.68986722720287928Ú`Ê¡”Š‚(:‰ ®öêÒý¶éÇ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚235.428' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.0856 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HÓãàã‹‚(:Ø ®öêÒ–”‘Ê disk.sda3.out‚ 31.8203125 disk.rbd0.out‚0 disk.rbd2.out‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0$ disk./.usageGB‚580.1191139221191 disk.sda3.in‚0 disk.rbd2.in‚0* proc.memory.percent‚17.081475257873535 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚99.95760917663574# proc.memory.rssMB‚ 2798.62890625 disk.dm-1.out‚ 31.8203125 disk.rbd1.in‚0 disk.drbd2165.out‚0 disk.dm-0.out‚3531.978515625# disk.nvme0n1.out‚3531.978515625 network.sent‚ 1.158644e+06 disk.rbd0.in‚0 disk.dm-0.in‚ 1767.66015625! disk.nvme0n1.in‚ 1767.66015625 cpu‚25.45529137654109) disk./.usagePercent‚38.93228048767101 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 31.8203125# disk.nvme0c0n1.in‚ 1767.66015625 network.recv‚344165 disk.rbd1.out‚3.390625% disk.nvme0c0n1.out‚3531.978515625% proc.memory.availableMB‚ 6.9453125\Å€”Œ‚(:‰ ½öêÒˆÚ¬Ç gpu.0.gpu‚86.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚244.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.9678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M\ ëã‚(:Ø ½öêÒ«¸§É disk.sda2.out‚0 disk.sda.in‚0" disk.dm-0.out‚3861.53662109375 disk.rbd0.in‚0 disk.rbd2.out‚0% proc.memory.availableMB‚ 6.6953125 network.sent‚ 1.22132e+06) disk./.usagePercent‚39.03674128992735 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.out‚32.28125 disk.sda.out‚32.28125# disk.nvme0c0n1.in‚ 1767.66015625 disk.dm-0.in‚ 1767.66015625$ memory_percent‚99.95913505554199 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚32.28125% disk.nvme0n1.out‚3861.53662109375 disk.sda1.in‚0# proc.memory.rssMB‚ 2798.62890625$ disk./.usageGB‚581.6756553649902 disk.drbd2165.out‚0 disk.sda3.in‚0* proc.memory.percent‚17.081475257873535 network.recv‚348380 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚3861.53662109375! disk.nvme0n1.in‚ 1767.66015625 disk.sda2.in‚0 disk.rbd0.out‚0 cpu‚25.454267523924777 disk.rbd1.out‚3.453125 disk.rbd2.in‚0þ¢ŸŸŽ‚(:” ÌöêÒ‡ÄìÉ gpu.0.gpu‚86.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚284.911' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.982200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ~”Ðé‚(:Þ ÌöêÒèéŽË disk.rbd2.out‚0 disk.sda1.out‚0 network.recv‚351642$ disk./.usageGB‚582.8820266723633 disk.rbd2.in‚0 disk.dm-1.in‚0 disk.sda.in‚0" disk.dm-0.out‚3954.21630859375& proc.memory.availableMB‚ 7.13671875 disk.sda3.in‚0 disk.sda3.out‚ 32.484375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚39.117701880578096' disk.nvme0c0n1.out‚3954.21630859375 disk.dm-0.in‚ 1769.43359375 disk.sda2.in‚0 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 1769.43359375 disk.sda1.in‚0$ memory_percent‚99.95644092559814 network.sent‚ 1.226438e+06 disk.dm-1.out‚ 32.484375 disk.sda.out‚ 32.484375* proc.memory.percent‚17.081475257873535 cpu‚25.453354197695724 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 1769.43359375 disk.sda2.out‚0 disk.rbd1.out‚3.453125% disk.nvme0n1.out‚3954.21630859375# proc.memory.rssMB‚ 2798.62890625l{Δ‚(:‰ ÛöêÒ²¿®Ç gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚133.042' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.6084 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ©êQÜ‘‚(:Ñ ÛöêÒÚÁþÉ disk.sda2.in‚0 disk.dm-1.out‚ 33.06640625 disk.sda3.out‚ 33.06640625% proc.memory.availableMB‚ 6.8984375$ disk./.usageGB‚583.0202369689941 disk.dm-0.out‚ 4697.140625 disk.sda3.in‚0) disk./.usagePercent‚39.12697728955116 disk.rbd2.out‚0 disk.rbd1.out‚3.53125 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 1770.6484375# proc.memory.rssMB‚ 2798.62890625 disk.rbd2.in‚0 disk.sda.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 1770.6484375 cpu‚25.452388885642115 network.sent‚ 1.231624e+06 disk.rbd0.in‚0 disk.dm-1.in‚0$ disk.nvme0c0n1.out‚ 4696.71484375 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚99.95789527893066* proc.memory.percent‚17.081475257873535 proc.cpu.threads‚17 network.recv‚354971 disk.nvme0n1.out‚ 4697.265625 disk.rbd0.out‚0 disk.sda.out‚ 33.06640625 disk.dm-0.in‚ 1770.6484375¿)Vªž’‚(:“ êöêÒîÕ†É gpu.0.gpu‚86.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚242.43' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.486000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o“,@í“‚(:â êöêÒ…„ÜÉ network.sent‚ 1.235593e+06 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1826.31640625 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.sda1.out‚0) disk./.usagePercent‚39.10800405917354 disk.sda.out‚ 33.43359375# disk.nvme0c0n1.in‚ 1826.31640625 disk.sda3.in‚0 disk.rbd0.in‚0 cpu‚25.451603242301232$ disk./.usageGB‚582.7375221252441 disk.sda2.in‚0 disk.dm-1.out‚ 33.43359375 disk.drbd2165.in‚0 disk.dm-0.in‚ 1826.31640625 disk.sda1.in‚0$ memory_percent‚99.95801448822021& proc.memory.availableMB‚ 6.87890625 disk.sda.in‚0% disk.nvme0n1.out‚6328.33544921875# proc.memory.rssMB‚ 2798.62890625 proc.cpu.threads‚17 disk.rbd1.out‚3.53125' disk.nvme0c0n1.out‚6328.33544921875 disk.sda3.out‚ 33.43359375 disk.rbd2.out‚0 network.recv‚357525 disk.dm-1.in‚0" disk.dm-0.out‚6328.33544921875 disk.rbd2.in‚0* proc.memory.percent‚17.081475257873535c¯¨¦2”- _wandbruntime‚1287  _runtime‚1287bî½DÅ•¿ & lr-AdamW/pg2‚0.0004820960512694913 trainer/global_step‚749 % _timestamp‚1.7843311269960651e+09 _runtime‚1287.581779456 _step‚57 & lr-AdamW/pg1‚0.0004820960512694913‰;Ä$É–Ã & lr-AdamW/pg1‚0.0004820960512694913 & lr-AdamW/pg2‚0.0004820960512694913 trainer/global_step‚749 % _timestamp‚1.7843311269960651e+09 _runtime‚1287.581779456 _step‚579Ø1ì2—- _wandbruntime‚1287  _runtime‚1287ë´¡E×'˜ÑO VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6451515 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213022411576133 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 699876691968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12742392 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚212416005087232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213117253341184 trainer/global_step‚749 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58355037458432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚212416005087232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1920372 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1920367 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4027226 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212323756396544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213117253341184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58354669367296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚212416031260672 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2107125 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213115881779200 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚1287.582308676 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6441544 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213115881779200 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12732415 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6290871 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820`$¾é¸ ™ÕO OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6441544 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 900073071616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6290871 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59254742438912 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 900109706240 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6441544 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213115881779200 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12732415 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6290871 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6451515 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213022411576133 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 699876691968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12742392 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚212416005087232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213117253341184 trainer/global_step‚749 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58355037458432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚212416005087232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1920372 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1920367 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4027226 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212323756396544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213117253341184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58354669367296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚212416031260672 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2107125 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213115881779200 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚1287.582308676 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4032485 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1288.997587998 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6299083 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213302586848949 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 700801727488 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12759090 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6450022 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚212601612746752 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 901288868864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6460007 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59333812734976 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6299077 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213396179113984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2109837 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212694005817344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 700948674229 trainer/global_step‚749 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213396179113984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58432155774976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6299083 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚213394807544832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚212694005817344 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1922915 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 699604073803 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1922910 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 % _timestamp‚1.7843311284121687e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995Šê€å&œßM PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1288.997587998 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6299083 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213302586848949 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 700801727488 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12759090 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6450022 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚212601612746752 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 901288868864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6460007 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59333812734976 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6299077 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213396179113984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2109837 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212694005817344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 700948674229 trainer/global_step‚749 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213396179113984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58432155774976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6299083 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚213394807544832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚212694005817344 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1922915 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 699604073803 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1922910 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 % _timestamp‚1.7843311284121687e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2109575 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚213301216820555 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚212694031990784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12759090 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12749099 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6299077 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6460007 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4032485 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0;‹Wv2- _wandbruntime‚1289  _runtime‚1289åµÉÑÿžù 1loss/SRqqt2/detection_loss‚0.7238238279481665 % _timestamp‚1.7843311284889698e+09 2loss/SRqqt1/assignment_loss‚1.5560631957061652 2loss/SRqqt2/assignment_loss‚1.3901776294596042 /loss/FRt1/detection_loss‚0.5938542256452684 .loss/FBt2/detection_loss‚2.150204911828041 _runtime‚1289.074067354 .loss/SRqqt1/detection_loss‚2.0218301116168 /loss/FBt1/detection_loss‚1.0304654802365463 0loss/FRt1/assignment_loss‚3.6362555516900215 1loss/FBt1/assignment_loss‚0.09945916591867326 %loss/total_loss‚17.21896031957041 epoch‚1 trainer/global_step‚749 _step‚60 /loss/FRt2/assignment_loss‚2.443963911085047 0loss/FBt2/assignment_loss‚0.1093814186945868 /loss/FRt2/detection_loss‚1.4634808897414906GØwŸý 2loss/SRqqt2/assignment_loss‚1.3901776294596042 /loss/FRt1/detection_loss‚0.5938542256452684 .loss/FBt2/detection_loss‚2.150204911828041 _runtime‚1289.074067354 .loss/SRqqt1/detection_loss‚2.0218301116168 /loss/FBt1/detection_loss‚1.0304654802365463 0loss/FRt1/assignment_loss‚3.6362555516900215 1loss/FBt1/assignment_loss‚0.09945916591867326 %loss/total_loss‚17.21896031957041 epoch‚1 trainer/global_step‚749 _step‚60 /loss/FRt2/assignment_loss‚2.443963911085047 0loss/FBt2/assignment_loss‚0.1093814186945868 /loss/FRt2/detection_loss‚1.4634808897414906 1loss/SRqqt2/detection_loss‚0.7238238279481665 % _timestamp‚1.7843311284889698e+09 2loss/SRqqt1/assignment_loss‚1.5560631957061652<ÄsV8” ‚(:‰ ùöêÒö×ÁÇ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚323.823' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.7646 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨3åJã¡‚(:Ø ùöêÒïÚ‚Ë disk.rbd0.out‚0& proc.memory.availableMB‚ 6.90234375 disk.sda.out‚ 33.7578125 disk.nvme0n1.in‚ 1827.1484375$ memory_percent‚99.95787143707275* proc.memory.percent‚17.081475257873535 disk.dm-1.out‚ 33.7578125 disk.rbd1.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2798.62890625 cpu‚25.45077219415149 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 1827.1484375& disk.nvme0c0n1.out‚6480.5439453125 disk.sda3.in‚0 disk.rbd2.out‚0 network.sent‚ 1.240602e+06 network.recv‚361174 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0$ disk./.usageGB‚582.6374549865723 disk.sda2.in‚0 disk.sda3.out‚ 33.7578125) disk./.usagePercent‚39.10128846953535 disk.rbd1.out‚3.59375 disk.sda.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 1827.1484375! disk.dm-0.out‚6480.5439453125 disk.rbd0.in‚0$ disk.nvme0n1.out‚6480.5439453125Íz+Ÿ¢‚(:” ˆ÷êÒº¸ÐÇ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚290.567' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.113400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&öØ£‚(:Í ˆ÷êÒ¢¾‘Ê disk.rbd1.out‚3.59375 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.dm-0.out‚ 6651.0078125 disk.nvme0n1.in‚ 1827.1484375! proc.memory.availableMB‚6.875" proc.memory.rssMB‚ 2798.6328125* proc.memory.percent‚17.081499099731445 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 34.03515625 disk.sda3.out‚ 34.03515625 proc.cpu.threads‚17 disk.rbd0.out‚0 cpu‚25.449858869543203 network.sent‚ 1.303297e+06 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 1827.1484375# disk.nvme0c0n1.out‚ 6651.0078125 disk.dm-0.in‚ 1827.1484375 disk.rbd0.in‚0 disk.sda2.out‚0! disk.nvme0n1.out‚ 6651.0078125) disk./.usagePercent‚39.09955453050594$ disk./.usageGB‚582.6116180419922 disk.dm-1.out‚ 34.03515625$ memory_percent‚99.95803833007812 network.recv‚365482 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd2.out‚0@’i៤‚(:” —÷êÒÝ¡çÇ gpu.0.gpu‚85.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚263.406' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.681200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·c>饂(:Þ —÷êÒ÷ŽæÉ disk.drbd2165.out‚0$ memory_percent‚99.95863437652588) disk./.usagePercent‚39.09954915434772 disk.rbd2.out‚0 disk.dm-1.out‚ 34.2734375 disk.rbd1.out‚3.59375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 34.2734375' disk.nvme0c0n1.out‚6664.74169921875" disk.dm-0.out‚6664.74169921875! disk.nvme0n1.in‚ 1828.11328125 disk.rbd2.in‚0 network.sent‚ 1.308272e+06 disk.sda3.in‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1828.11328125* proc.memory.percent‚17.081499099731445$ disk./.usageGB‚582.6115379333496 disk.sda1.out‚0 cpu‚25.448758811968002 proc.cpu.threads‚17 network.recv‚369224% disk.nvme0n1.out‚6664.74169921875 disk.sda3.out‚ 34.2734375 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 1828.11328125" proc.memory.rssMB‚ 2798.6328125 disk.sda1.in‚0 disk.rbd1.in‚0& proc.memory.availableMB‚ 6.77734375JŠdÙ¦‚(:’ ¦÷êÒЯ´Ç gpu.0.gpu‚83.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚301.52' gpu.0.eÛß:ÄnforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.30399999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09ƒYᧂ(:Ö ¦÷êÒÞáöÉ network.recv‚372661# disk.nvme0c0n1.in‚ 1876.95703125" proc.memory.rssMB‚ 2798.6328125 disk.rbd2.in‚0 disk.rbd1.out‚3.671875 disk.drbd2165.out‚0 disk.sda3.out‚34.6875 disk.sda.in‚0 disk.rbd2.out‚0$ memory_percent‚99.95944499969482 network.sent‚ 1.313494e+06 disk.sda2.out‚0# disk./.usageGB‚582.611442565918 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda.out‚34.6875 disk.drbd2165.in‚0% disk.nvme0n1.out‚6676.02685546875 disk.sda3.in‚0* proc.memory.percent‚17.081499099731445' disk.nvme0c0n1.out‚6676.02685546875 disk.dm-0.in‚ 1876.95703125" disk.dm-0.out‚6676.02685546875 disk.sda1.out‚0& proc.memory.availableMB‚ 6.64453125 disk.dm-1.out‚34.6875 disk.sda1.in‚0 disk.rbd0.out‚0 cpu‚25.447848358289413* disk./.usagePercent‚39.099542754159366 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1876.95703125 disk.rbd0.in‚0 proc.cpu.threads‚17·{f’¨‚(:‡ µ÷êÒ€¢ÅÇ gpu.0.gpu‚89.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚261.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»;ë©‚(:à µ÷êÒݳ»É disk.rbd0.in‚0 disk.rbd2.out‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1876.95703125 disk.dm-0.in‚ 1876.95703125 disk.sda1.in‚0" proc.memory.rssMB‚ 2798.6328125 network.recv‚376364 disk.rbd2.in‚0 disk.dm-1.out‚ 35.29296875 disk.sda.in‚0 disk.drbd2165.out‚0 cpu‚25.446980825984586 proc.cpu.threads‚17* disk./.usagePercent‚39.115592378495954 disk.dm-1.in‚0 disk.rbd1.out‚3.671875" disk.dm-0.out‚6955.55224609375 disk.sda1.out‚0 network.sent‚ 1.31838e+06% disk.nvme0n1.out‚6955.55224609375* proc.memory.percent‚17.081499099731445 disk.rbd1.in‚0 disk.sda.out‚ 35.29296875 disk.sda3.out‚ 35.29296875% proc.memory.availableMB‚ 6.6171875$ disk./.usageGB‚582.8505935668945 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 1876.95703125 disk.sda3.in‚0# memory_percent‚99.9596118927002 disk.rbd0.out‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚6955.55224609375’¥”ª‚(:‰ Ä÷êÒ«ÇšÇ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚285.114' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.0228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ#Ý«‚(:Ò Ä÷êÒù™É network.sent‚ 1.323362e+06 network.recv‚380005 disk.sda2.in‚0) disk./.usagePercent‚39.11559161047335 disk.dm-0.in‚ 1877.921875 disk.dm-0.out‚6977.654296875 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.rbd2.in‚0 disk.sda.in‚0" proc.memory.rssMB‚ 2798.6328125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd1.in‚0% disk.nvme0c0n1.out‚6977.654296875 cpu‚25.446304255383463 disk.nvme0n1.in‚ 1877.921875 disk.rbd2.out‚0 disk.dm-1.out‚ 35.5859375& proc.memory.availableMB‚ 6.61328125 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 1877.921875# memory_percent‚99.9596357345581* proc.memory.percent‚17.081499099731445$ disk./.usageGB‚582.8505821228027 disk.sda.out‚ 35.5859375# disk.nvme0n1.out‚6977.654296875 disk.sda3.out‚ 35.5859375 disk.rbd0.out‚0 disk.rbd1.out‚3.734375MÙÅ2¬- _wandbruntime‚1371  _runtime‚1371|vŠ¿Ç­Á ' lr-AdamW/pg1‚0.00048198022387763113 ' lr-AdamW/pg2‚0.00048198022387763113 trainer/global_step‚799 % _timestamp‚1.7843312106241052e+09 _runtime‚1371.209955593 _step‚61 w^"ˮŠtrainer/global_step‚799 % _timestamp‚1.7843312106241052e+09 _runtime‚1371.209955593 _step‚61 ' lr-AdamW/pg1‚0.00048198022387763113 ' lr-AdamW/pg2‚0.00048198022387763113=9r@2¯- _wandbruntime‚1371  _runtime‚1371#J§¬Ö'°ÐO YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚226316041592832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2047522 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 959493993984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚226217845305344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚1371.213795112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6867296 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4292215 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _step‚62 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227062234215424 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13578444 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6877267 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226962737598041 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13568467 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6877267 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6701177 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63188483745280 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2047517 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚226316041592832 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6701171 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4291946 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 747538011136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚226316067766272 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚227063605777408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226316067766272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36634624 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚227063605777408 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2244693 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820tÝŸÚ'±ÔO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227062234215424 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13578444 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6877267 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226962737598041 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13568467 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6877267 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6701177 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63188483745280 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2047517 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚226316041592832 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6701171 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4291946 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 747538011136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚226316067766272 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚227063605777408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226316067766272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36634624 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚227063605777408 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2244693 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚226316041592832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2047522 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 959493993984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚226217845305344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚1371.213795112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6867296 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4292215 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _step‚62>¶F_2²- _wandbruntime‚1372  _runtime‚1372ÒŽQ3³ÛM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6709377 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 747117658112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 747117658112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13595142 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6875774 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚226594042322944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6875774 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 745841186911 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13585151 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2247186 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226495727083520 _runtime‚1372.625470667 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 748463053824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13595142 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62306476158976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6885759 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6709377 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6709383 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843312120397666e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚227341159981056 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2247448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch}4ŒF®_end/reserved_bytes.large_pool.peak‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226594068496384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62306844250112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 960676673024 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 747185787337 _step‚63 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4297513 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4297246 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚227242912870857 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227241542842463 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 trainer/global_step‚799x.úå&´ßM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2050060 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚227342531550208 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 748463053824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13595142 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62306476158976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6885759 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6709377 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6709383 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843312120397666e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚227341159981056 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2247448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226594068496384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62306844250112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 960676673024 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 747185787337 _step‚63 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4297513 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4297246 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚227242912870857 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227241542842463 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 trainer/global_step‚799 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6709377 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 747117658112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 747117658112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13595142 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6875774 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚226594042322944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6875774 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 745841186911 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13585151 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2247186 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226495727083520 _runtime‚1372.625470667 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B½Ž| 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.06064533442258835 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4337500035762787 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.2769918739795685 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.0967622846364975 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.16325496137142181 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3153972923755646 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9952781200408936 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.22895857691764832 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.007130124606192112 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9883924126625061 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.986586332321167 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.4921875 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 epoch‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.22901932895183563 particle/accuracy_4_of_5‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.09182138741016388 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 /particle/accuracy_3_of_4‚0.9970238208770752 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9697766304016113 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2656259536743164 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2870370149612427 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.059512894600629807 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.993054986000061 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.005642869509756565 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2135416716337204 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 7"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.27696493268013 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.09898604452610016 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.274585485458374 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0356430746614933 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7validation_average_jet_accuracy‚0.18795187771320343 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.29103192687034607 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1606231927871704 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6924791932106018 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.03666270896792412 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 trainer/global_step‚799 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4377986788749695 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9959514141082764 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.15708507597446442 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.35644280910491943 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4164424538612366 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.26281824707984924 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4381290078163147 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 +jet/accuracy_2_of_2‚0.08041375130414963 (particle/accuracy‚0.8090704083442688 +particle/specificity‚0.8735691905021667 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.885023295879364 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.16174986958503723 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.31581276655197144 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_2_of_4‚0.33563217520713806 &particle/f_score‚0.563223123550415 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.43892064690589905 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.04738764092326164 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3253158926963806 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19572366774082184 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9895137548446655 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.44084158539772034 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.04630814120173454 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.13695597648620605 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2799077332019806 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.25345519185066223 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 particle/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9756010174751282 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0005537098622880876 $ _timestamp‚1.784331240385615e+09 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23202522099018097 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.2826850414276123 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09656364470720291 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.048366256058216095 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.25165945291519165 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9921478033065796 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 +jet/accuracy_1_of_2‚0.48720890283584595 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8143104314804077 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9676709175109863 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.2753307521343231 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.994735062122345 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4594687819480896 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17618772387504578 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.25189051032066345 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.13292567431926727 *jet/accuracy_1_of_1‚0.1472509801387787 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9759637117385864 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_3‚1 /particle/accuracy_4_of_4‚0.9545771479606628 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5694444179534912 *jet/accuracy_1_of_4‚0.6089423298835754 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.2835868299007416 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9863166809082031 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6876795291900635 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8526696562767029 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9696178436279297 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2795366644859314 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.036281000822782516 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3168356716632843 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.45908504724502563 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.11424990743398666 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.42553651332855225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16131345927715302 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9868519902229309 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.05694591999053955 particle/accuracy_3_of_5‚1 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.046317797154188156 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7969145774841309 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9872292280197144 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.296875 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.27635541558265686 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.014201467856764793 /particle/accuracy_3_of_3‚0.9994933605194092 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.27132394909858704 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_5‚1 5 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.99150151014328 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.45487266778945923 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03676928952336311 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.048366256058216095 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9950970411300659 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5694444179534912 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 jet/accuracy_3_of_5‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9392358064651489 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07961729913949966 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5675737261772156 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6213151812553406 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.32530441880226135 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.10868679732084274 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.27646079659461975 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.27132394909858704 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23780643939971924 ,jet/accuracy_3_of_3‚0.045109815895557404 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.31564217805862427 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 +jet/accuracy_3_of_4‚0.04521290585398674 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 _runtime‚1400.971544408 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4731091558933258 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.26935598254203796 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0009718172950670123 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8481117486953735 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6627658605575562 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6890400052070618 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.38609573245048523 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.10805622488260269 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9960317015647888 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 jet/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4625592529773712 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17502395808696747 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.40509259700775146 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13292567431926727 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08174945414066315 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.12783706188201904 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.26047906279563904 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 *jet/accuracy_1_of_3‚0.6524582505226135 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.41626104712486267 +particle/sensitivity‚0.5732411742210388 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.422843337059021 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.007501085754483938 particle/accuracy_1_of_3‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9846486449241638 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8657407164573669 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6934996247291565 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.060361750423908234 *loss/val_total_loss‚15.746235239602001 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9830721616744995 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18355277180671692 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.0716666653752327 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9952781200408936 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2769918739795685 particle/accuracy_1_of_2‚1 jet/accuracy_1_of_5‚1 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2227427214384079 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.31701740622520447 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.010978731326758862 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9844617247581482 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _step‚65 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.28637558221817017 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9980952143669128¤èÿm™¾’| 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.060361750423908234 *loss/val_total_loss‚15.746235239602001 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.983®ã‘|=0721616744995 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18355277180671692 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.0716666653752327 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9952781200408936 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2769918739795685 particle/accuracy_1_of_2‚1 jet/accuracy_1_of_5‚1 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2227427214384079 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.31701740622520447 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.010978731326758862 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9844617247581482 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _step‚65 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.28637558221817017 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9980952143669128 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.06064533442258835 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4337500035762787 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.2769918739795685 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.0967622846364975 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.16325496137142181 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3153972923755646 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9952781200408936 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.22895857691764832 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.007130124606192112 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9883924126625061 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.986586332321167 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.4921875 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 epoch‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.22901932895183563 particle/accuracy_4_of_5‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.09182138741016388 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 /particle/accuracy_3_of_4‚0.9970238208770752 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9697766304016113 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2656259536743164 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2870370149612427 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.059512894600629807 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.993054986000061 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.005642869509756565 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2135416716337204 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 7"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.27696493268013 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.09898604452610016 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.274585485458374 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0356430746614933 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7validation_average_jet_accuracy‚0.18795187771320343 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.29103192687034607 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1606231927871704 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6924791932106018 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.03666270896792412 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 trainer/global_step‚799 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4377986788749695 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9959514141082764 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.15708507597446442 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.35644280910491943 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4164424538612366 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.26281824707984924 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4381290078163147 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 +jet/accuracy_2_of_2‚0.08041375130414963 (particle/accuracy‚0.8090704083442688 +particle/specificity‚0.8735691905021667 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.885023295879364 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.16174986958503723 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.31581276655197144 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_2_of_4‚0.33563217520713806 &particle/f_score‚0.563223123550415 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.43892064690589905 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.04738764092326164 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3253158926963806 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19572366774082184 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9895137548446655 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.44084158539772034 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.04630814120173454 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.13695597648620605 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2799077332019806 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.25345519185066223 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 particle/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9756010174751282 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0005537098622880876 $ _timestamp‚1.784331240385615e+09 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23202522099018097 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.2826850414276123 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09656364470720291 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.048366256058216095 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.25165945291519165 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9921478033065796 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 +jet/accuracy_1_of_2‚0.48720890283584595 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8143104314804077 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9676709175109863 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.2753307521343231 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.994735062122345 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4594687819480896 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17618772387504578 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.25189051032066345 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.13292567431926727 *jet/accuracy_1_of_1‚0.1472509801387787 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9759637117385864 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_3‚1 /particle/accuracy_4_of_4‚0.9545771479606628 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5694444179534912 *jet/accuracy_1_of_4‚0.6089423298835754 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.2835868299007416 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9863166809082031 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6876795291900635 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8526696562767029 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9696178436279297 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2795366644859314 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.036281000822782516 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3168356716632843 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.45908504724502563 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.11424990743398666 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.42553651332855225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16131345927715302 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9868519902229309 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.05694591999053955 particle/accuracy_3_of_5‚1 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.046317797154188156 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7969145774841309 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9872292280197144 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.296875 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.27635541558265686 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.014201467856764793 /particle/accuracy_3_of_3‚0.9994933605194092 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.27132394909858704 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_5‚1 5 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.99150151014328 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.45487266778945923 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03676928952336311 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.048366256058216095 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9950970411300659 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5694444179534912 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 jet/accuracy_3_of_5‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9392358064651489 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07961729913949966 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5675737261772156 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6213151812553406 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.32530441880226135 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.10868679732084274 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.27646079659461975 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.27132394909858704 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23780643939971924 ,jet/accuracy_3_of_3‚0.045109815895557404 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.31564217805862427 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 +jet/accuracy_3_of_4‚0.04521290585398674 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 _runtime‚1400.971544408 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4731091558933258 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.26935598254203796 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0009718172950670123 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8481117486953735 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6627658605575562 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6890400052070618 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.38609573245048523 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.10805622488260269 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9960317015647888 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 jet/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4625592529773712 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17502395808696747 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.40509259700775146 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13292567431926727 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08174945414066315 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.12783706188201904 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.26047906279563904 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 *jet/accuracy_1_of_3‚0.6524582505226135 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.41626104712486267 +particle/sensitivity‚0.5732411742210388 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.422843337059021 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.007501085754483938 particle/accuracy_1_of_3‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9846486449241638 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8657407164573669 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6934996247291565 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0A /Ý”¿‚(:‰ ñ÷êÒϨûÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚260.702' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.1404 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®ÓgíãÀ‚(:Ø ñ÷êÒÉ% disk.nvme0n1.out‚8119.17431640625 disk.dm-1.in‚0 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 1934.296875# memory_percent‚88.1157636642456# proc.memory.rssMB‚ 2837.72265625 disk.sda3.out‚ 38.5703125) disk./.usagePercent‚39.17907098265604 proc.cpu.threads‚17 network.recv‚391334 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚3.8125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 1934.296875" disk.dm-0.out‚8119.17431640625 disk.nvme0n1.in‚ 1934.296875 disk.sda2.out‚0 disk.sda.in‚0' disk.nvme0c0n1.out‚8119.17431640625 cpu‚25.48159883437001 network.sent‚ 1.396054e+06 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd2.out‚0$ disk./.usageGB‚583.7964706420898 disk.dm-1.out‚ 38.5703125 disk.sda.out‚ 38.5703125) proc.memory.availableMB‚ 1947.11328125) proc.memory.percent‚17.32008457183838 disk.drbd2165.in‚0 disk.sda1.out‚0c÷dšŸÁ‚(:” €øêÒ™äÈ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚287.087' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.417399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QÍÙ\í‚(:â €øêÒŽãÉ disk.drbd2165.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 1935.12890625 disk.rbd2.in‚0 cpu‚25.480397040881048 disk.drbd2165.in‚0* disk./.usagePercent‚39.179136264577274 disk.rbd1.out‚3.875) proc.memory.availableMB‚ 1946.58984375 disk.rbd0.in‚0) proc.memory.percent‚17.32487678527832 proc.cpu.threads‚17 network.sent‚ 1.454231e+06$ disk./.usageGB‚583.7974433898926 disk.dm-1.out‚ 38.90234375' disk.nvme0c0n1.out‚9013.98486328125" disk.dm-0.out‚9013.98486328125 disk.sda1.in‚0$ memory_percent‚88.11895847320557 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 1935.12890625 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 38.90234375 disk.rbd2.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚9013.98486328125" proc.memory.rssMB‚ 2838.5078125 network.recv‚395847! disk.nvme0n1.in‚ 1935.12890625 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda.out‚ 38.90234375z‡ŸÃ‚(:” øêÒ¿ÔýÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚298.392' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.678399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0' ¤ãÄ‚(:Ø øêÒ…ÕÚÊ! disk.dm-0.out‚9017.0107421875 disk.sda3.out‚ 39.2265625 network.recv‚399178* disk./.usagePercent‚39.179038469699194$ disk./.usageGB‚583.7959861755371 disk.sda.out‚ 39.2265625# proc.memory.rssMB‚ 2837.70703125) proc.memory.percent‚17.31998920440674 disk.sda.in‚0$ disk.nvme0n1.out‚9017.0107421875 disk.rbd2.out‚0 disk.rbd1.in‚0" memory_percent‚88.114333152771 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 39.2265625! disk.nvme0n1.in‚ 1935.12890625 disk.sda1.in‚0 disk.sda3.in‚0 cpu‚25.47919856334537 network.sent‚ 1.4591e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚9017.0107421875 disk.rbd1.out‚3.875# disk.nvme0c0n1.in‚ 1935.12890625 disk.dm-0.in‚ 1935.12890625 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1947.34765625õYyžÅ‚(:“ žøêÒàþÇ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚304.756' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.95119999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+»ÇèãÆ‚(:Ø žøêÒ¾ÉšÊ disk.rbd2.in‚0 disk.sda.in‚0 disk.drbd2165.out‚0) proc.memory.percent‚17.32487678527832 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 1946.6015625" proc.memory.rssMB‚ 2838.5078125 disk.sda3.out‚ 39.49609375( disk./.usagePercent‚39.1790863431081$ memory_percent‚88.11888694763184" disk.dm-0.out‚9025.60888671875 disk.rbd0.out‚0 disk.drbd2165.in‚0 cpu‚25.47807625792089 disk.nvme0c0n1.in‚ 1936.09375 disk.sda.out‚ 39.49609375 network.sent‚ 1.464099e+06' disk.nvme0c0n1.out‚9025.60888671875% disk.nvme0n1.out‚9025.60888671875 disk.rbd0.in‚0 disk.rbd2.out‚0 disk.dm-1.out‚ 39.49609375 disk.rbd1.out‚3.953125 network.recv‚402405 disk.nvme0n1.in‚ 1936.09375 disk.sda3.in‚0 disk.dm-1.in‚0$ disk./.usageGB‚583.7966995239258 disk.dm-0.in‚ 1936.09375³Å ŸÇ‚(:” ­øêÒªçÇÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚267.174' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.434799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HÛô`æÈ‚(:Û ­øêÒª¦¢Ê disk.rbd1.out‚3.953125 network.recv‚405958 disk.dm-0.in‚ 1992.8515625$ disk.nvme0n1.out‚9029.4931640625) proc.memory.availableMB‚ 1946.30078125 disk.sda3.out‚ 39.9921875" proc.memory.rssMB‚ 2838.5078125) proc.memory.percent‚17.32487678527832$ disk./.usageGB‚583.7964859008789! disk.dm-0.out‚9029.4931640625 disk.sda1.in‚0 cpu‚25.476956443363505 network.sent‚ 1.469124e+06" disk.nvme0c0n1.in‚ 1992.8515625 disk.sda1.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.17907200668618 disk.sda.out‚ 39.9921875& disk.nvme0c0n1.out‚9029.4931640625 disk.nvme0n1.in‚ 1992.8515625$ memory_percent‚88.12072277069092 disk.sda2.in‚0 disk.dm-1.out‚ 39.9921875 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0϶ý¼”É‚(:‰ ¼øêÒ ä¯È gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚253.693' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´q¯ôäÊ‚(:Ù ¼øêÒôýÉ$ disk./.usageGB‚583.7964515686035 disk.drbd2165.in‚0 disk.dm-1.out‚ 40.1640625( proc.memory.availableMB‚ 1946.3046875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda2.in‚0$ memory_percent‚88.12069892883301 network.sent‚ 1.474109e+06 disk.dm-0.in‚ 1993.68359375 disk.sda3.out‚ 40.1640625 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 1993.68359375 disk.sda.out‚ 40.1640625" proc.memory.rssMB‚ 2838.5078125 cpu‚25.47603653966856% disk.nvme0c0n1.out‚9039.751953125 disk.sda3.in‚0) proc.memory.percent‚17.32487678527832) disk./.usagePercent‚39.17906970261837 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.rbd1.in‚0# disk.nvme0n1.out‚9039.751953125 disk.drbd2165.out‚0 disk.dm-0.out‚9039.751953125! disk.nvme0n1.in‚ 1993.68359375 disk.rbd2.out‚0 network.recv‚409235 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚4.015625Nïª2Ë- _wandbruntime‚1488  _runtime‚1488ß÷’œÅÌ¿ trainer/global_step‚849 % _timestamp‚1.7843313283199232e+09 _runtime‚1488.905004939 _step‚66 & lr-AdamW/pg1‚0.0004818489339789779 & lr-AdamW/pg2‚0.0004818489339789779€ÉÍà trainer/global_step‚849 % _timestamp‚1.7843313283199232e+09 _runtime‚1488.905004939 _step‚66 & lr-AdamW/pg1‚0.0004818489339789779 & lr-AdamW/pg2‚0.0004818489339789779BÒvÏ2Î- _wandbruntime‚1488  _runtime‚1488Ø~eÏÓO dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚66151790473216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚244734801683991 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7352030 _runtime‚1488.906519568 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚244044085706752 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7352030 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 799696553984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14595507 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67178569669120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚244044085706752 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 798351164928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 798328565525 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚226 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2415170 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7243483 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 796983965207 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7243477 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7243483 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244842410698240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚849 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚244843782260736 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7243477 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67178970200064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 798351164928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244842410698240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚244044059533312 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1027179726848 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚243937843146752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843313283211622e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚244044059533312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4598755 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2183590 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7362002 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2415391 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚243937817718784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚244736171712277 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2183585 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 799696553984 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244843782260736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4598981 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1027147287040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440§FWzÝ'Ð×O `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66151422382080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚244736171712277 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2183585 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 799696553984 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244843782260736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4598981 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1027147287040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚66151790473216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚244734801683991 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7352030 _runtime‚1488.906519568 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚244044085706752 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7352030 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 799696553984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14595507 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67178569669120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚244044085706752 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 798351164928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 798328565525 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚226 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2415170 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7243483 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 796983965207 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7243477 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7243483 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244842410698240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚849 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚244843782260736 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7243477 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67178970200064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 798351164928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244842410698240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚244044059533312 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1027179726848 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚243937843146752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843313283211622e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚244044059533312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4598755 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2183590 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7362002 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2415391 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚243937817718784 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚244322086436864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚244322060263424 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7370494 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚211 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4604034 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚244322086436864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚244322060263424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚245121336463872 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7251689 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _step‚68 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7251683 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 800621596672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67257671066624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14622183 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2418117 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 797907431453 trainer/global_step‚849 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14622183 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 799276200448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2186133 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66229276880896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843313297674503e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 799252031883 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚1490.353035757 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚245122708033536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚244215699496960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872°¸c\ç&ÓáM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚245122708033536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚244215699496960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚244322086436864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚244322060263424 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7370494 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚211 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4604034 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚244322086436864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚244322060263424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚245121336463872 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7251689 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _step‚68 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7251683 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 800621596672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67257671066624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14622183 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2418117 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 797907431453 trainer/global_step‚849 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14622183 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 799276200448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2186133 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66229276880896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843313297674503e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 799252031883 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚1490.353035757 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0D‰sY&2Ô- _wandbruntime‚1490  _runtime‚1490yŒ}ùÕý 0loss/FRt1/assignment_loss‚3.3352661058802298 2loss/SRqqt2/assignment_loss‚1.1982439443247304 trainer/global_step‚849 _runtime‚ 1490.39567582 0loss/FRt2/assignment_loss‚2.2855538353193454 1loss/SRqqt1/detection_loss‚1A¶³ .3893826874819668 .loss/FBt1/detection_loss‚0.689877167392873 &loss/total_loss‚15.162242531877087 1loss/SRqqt1/assignment_loss‚1.247419525464677 1loss/FBt1/assignment_loss‚0.20064227990998026 1loss/FBt2/assignment_loss‚0.14080233967412223 1loss/SRqqt2/detection_loss‚0.7424179495534017 epoch‚2 _step‚69 /loss/FRt1/detection_loss‚0.6383414044467725 /loss/FRt2/detection_loss‚1.4392327055083864 /loss/FBt2/detection_loss‚1.8550625869206019 % _timestamp‚1.7843313298104172e+09æ¿ÏÖ epoch‚2 _step‚69 /loss/FRt1/detection_loss‚0.6383414044467725 /loss/FRt2/detection_loss‚1.4392327055083864 /loss/FBt2/detection_loss‚1.8550625869206019 % _timestamp‚1.7843313298104172e+09 0loss/FRt1/assignment_loss‚3.3352661058802298 2loss/SRqqt2/assignment_loss‚1.1982439443247304 trainer/global_step‚849 _runtime‚ 1490.39567582 0loss/FRt2/assignment_loss‚2.2855538353193454 1loss/SRqqt1/detection_loss‚1.3893826874819668 .loss/FBt1/detection_loss‚0.689877167392873 &loss/total_loss‚15.162242531877087 1loss/SRqqt1/assignment_loss‚1.247419525464677 1loss/FBt1/assignment_loss‚0.20064227990998026 1loss/FBt2/assignment_loss‚0.14080233967412223 1loss/SRqqt2/detection_loss‚0.7424179495534017E‹Znž×‚(:“ ËøêÒ„ÿ¸É gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚296.857' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.37140000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{äŒàØ‚(:Õ ËøêÒüÜìÊ disk.dm-0.out‚ 9047.8828125! disk.nvme0n1.out‚ 9047.8828125 disk.rbd2.out‚0$ memory_percent‚88.12191486358643 disk.drbd2165.out‚0) disk./.usagePercent‚39.17908301501015 disk.sda1.in‚0 disk.dm-1.out‚ 40.45703125 disk.rbd1.out‚4.015625$ disk./.usageGB‚583.7966499328613 disk.sda3.out‚ 40.45703125 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 1994.6484375# disk.nvme0c0n1.out‚ 9047.8828125 cpu‚25.475094544536837 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 40.45703125) proc.memory.availableMB‚ 1946.10546875 network.recv‚412636 disk.sda3.in‚0" proc.memory.rssMB‚ 2838.5078125 proc.cpu.threads‚17 disk.dm-0.in‚ 1994.6484375 disk.nvme0n1.in‚ 1994.6484375 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.drbd2165.in‚0) proc.memory.percent‚17.32487678527832 network.sent‚ 1.479102e+06 disk.sda2.in‚0 disk.rbd1.in‚0ÿÏo’Ù‚(:‡ ÚøêÒ­°ªÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚267.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“<%ÌèÚ‚(:Ý ÚøêÒÿþÉ disk.rbd1.out‚4.09375' disk.nvme0c0n1.out‚9056.22802734375 cpu‚25.474068493146486# disk./.usageGB‚583.796573638916 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 40.83984375' proc.memory.availableMB‚ 1946.109375" proc.memory.rssMB‚ 2838.5078125 network.recv‚416702 disk.rbd0.out‚0% disk.nvme0n1.out‚9056.22802734375 network.sent‚ 1.542192e+06 disk.sda3.in‚0 disk.dm-1.out‚ 40.83984375 disk.sda1.in‚0$ memory_percent‚88.12189102172852) disk./.usagePercent‚39.17907789485947 disk.sda3.out‚ 40.83984375 disk.rbd2.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 1994.6484375 disk.drbd2165.out‚0" disk.dm-0.out‚9056.22802734375 disk.nvme0n1.in‚ 1994.6484375) proc.memory.percent‚17.32487678527832 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda.in‚0" disk.nvme0c0n1.in‚ 1994.6484375 disk.sda1.out‚0#â¿ÀŸÛ‚(:” éøêÒ˜èôÈ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚280.116' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.023199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–ÆPOêÜ‚(:ß éøêÒÁ¢÷É disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 41.1171875 disk.rbd0.in‚0 network.recv‚420399 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 41.1171875 disk.rbd1.in‚0' disk.nvme0c0n1.out‚9066.31201171875) proc.memory.percent‚17.32487678527832 disk.rbd1.out‚4.09375 disk.drbd2165.in‚0 disk.dm-0.in‚ 2043.20703125 disk.rbd2.in‚0# memory_percent‚88.1221055984497) disk./.usagePercent‚39.17908915919097! disk.nvme0n1.in‚ 2043.20703125 disk.sda.in‚0 disk.sda.out‚ 41.1171875 disk.rbd0.out‚0$ disk./.usageGB‚583.7967414855957" disk.dm-0.out‚9066.31201171875% disk.nvme0n1.out‚9066.31201171875 disk.rbd2.out‚0) proc.memory.availableMB‚ 1946.07421875 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 2043.20703125" proc.memory.rssMB‚ 2838.5078125 cpu‚25.472873736667598 network.sent‚ 1.547276e+06ÝUAjŸÝ‚(:” øøêÒ‘ÍãÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚292.642' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.528400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“˜˜ßÞ‚(:Ô øøêÒ戬Ê disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.out‚4.21875 proc.cpu.threads‚17# disk./.usageGB‚583.796688079834 disk.nvme0n1.in‚ 2044.171875 disk.drbd2165.in‚0$ disk.nvme0c0n1.out‚ 9078.98046875" disk.nvme0n1.out‚ 9078.98046875 disk.sda3.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 network.sent‚ 1.552147e+06 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd1.in‚0 network.recv‚423773 disk.drbd2165.out‚0 disk.dm-0.out‚ 9078.98046875( proc.memory.availableMB‚ 1946.6328125) proc.memory.percent‚17.32490062713623$ memory_percent‚88.11869621276855# proc.memory.rssMB‚ 2838.51171875! disk.nvme0c0n1.in‚ 2044.171875 disk.dm-0.in‚ 2044.171875 disk.sda1.in‚0 disk.sda3.out‚ 41.34765625 disk.rbd0.in‚0 disk.rbd2.out‚0 disk.dm-1.out‚ 41.34765625* disk./.usagePercent‚39.179085575085494 disk.sda.out‚ 41.34765625 cpu‚25.472163457234387Ž’‹”ß‚(:‰ ‡ùêÒªÏ®Ç gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚280.902' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ꂺ}åà‚(:Ú ‡ùêÒä¾ÞÉ disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 41.58984375 disk.drbd2165.in‚0 disk.sda3.out‚ 41.58984375 disk.rbd2.in‚0 network.sent‚ 1.557183e+06 disk.sda1.out‚0 disk.rbd0.out‚0* disk./.usagePercent‚39.179099143484805$ disk./.usageGB‚583.7968902587891% disk.nvme0n1.out‚9084.46044921875 disk.sda1.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2838.51171875 cpu‚25.471185609025085 disk.nvme0n1.in‚ 2044.171875 disk.rbd1.in‚0$ memory_percent‚88.11874389648438 proc.cpu.threads‚17 disk.rbd1.out‚4.21875 disk.sda.out‚ 41.58984375 disk.drbd2165.out‚0 disk.dm-0.in‚ 2044.171875) proc.memory.percent‚17.32490062713623 disk.rbd2.out‚0 disk.sda.in‚0" disk.dm-0.out‚9084.46044921875' disk.nvme0c0n1.out‚9084.46044921875 disk.rbd0.in‚0 network.recv‚427256 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 2044.171875$ proc.memory.availableMB‚1946.625º°%ô2á- _wandbruntime‚1572  _runtime‚1572‘àfÇâÁ ' lr-AdamW/pg1‚0.00048170219000603665 ' lr-AdamW/pg2‚0.00048170219000603665 trainer/global_step‚899 % _timestamp‚1.7843314115343666e+09 _runtime‚1572.119460399 _step‚70ðäÉhËãÅ ' lr-AdamW/pg1‚0.00048170219000603665 ' lr-AdamW/pg2‚0.00048170219000603665 trainer/global_step‚899 % _timestamp‚1.7843314115343666e+09 _runtime‚1572.119460399 _step‚70F&)Ž+2ä- _wandbruntime‚1572  _runtime‚1572ÄUR.Ù'åÓO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚257831906627584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71113725344768 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70025742766080 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4864859 trainer/global_step‚899 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2553874 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7653783 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7777782 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7653777 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2310735 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70026110857216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1087579950592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258676497734185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚258788763134464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1087614487552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15431559 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 846012484608 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 846012484608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2554119 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 844565678633 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7777782 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 844667095552 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71113322716672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚257944122212352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15441537 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚71 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7787754 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚257944096038912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258675127705899 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 843221078315 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2310740 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7787754 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4864609 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚257944122212352 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 844667095552 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7653783 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚1572.120928951 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚258788763134464 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7777782 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 844667095552 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71113322716672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚257944122212352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15441537 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚71 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7787754 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚257944096038912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258675127705899 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 843221078315 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2310740 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7787754 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4864609 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚257944122212352 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 844667095552 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7653783 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚1572.120928951 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚258788763134464 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4864859 trainer/global_step‚899 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2553874 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7653783 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7777782 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7653777 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2310735 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70026110857216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1087579950592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258676497734185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995G‚Uws2ç- _wandbruntime‚1573  _runtime‚1573fÙ7R èÝM % _timestamp‚1.7843314129629312e+09 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1088784684544 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4870135 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259067688900096 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2313278 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258222096769024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚258953932950321 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7786260 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15458235 A8DeviceStatsMonitor.on_train_batû9UbÖch_end/active.all.current‚9992 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2556852 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70103597264896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚258222122942464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1088819214336 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 846937527296 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15458235 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2556620 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71192416479232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7786260 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7661983 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚258109788405760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259069060469760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258222122942464 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚72 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚899 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4869898 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71192013858304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258955302978719 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7796246 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚258222096769024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529792oÛ‰ìç&éáM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1088819214336 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 846937527296 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15458235 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2556620 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71192416479232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7786260 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7661983 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚258109788405760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259069060469760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258222122942464 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚72 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚899 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4869898 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71192013858304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258955302978719 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7796246 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚258222096769024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529792 % _timestamp‚1.7843314129629312e+09 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1088784684544 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4870135 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259067688900096 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2313278 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258222096769024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚258953932950321 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7786260 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15458235 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2556852 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70103597264896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚258222122942464H¡; 2ê- _wandbruntime‚1573  _runtime‚1573¾ °ëþ 2loss/SRqqt2/assignment_loss‚1.1687392676925015 /loss/FRt2/detection_loss‚1.4413966486035295 &loss/total_loss‚14.736564866916158 epoch‚2 % _timestamp‚1.7843314130192788e+09 _step‚73 0loss/FRt1/assignment_loss‚3.4550086140503615 2loss/SRqqt1/assignment_loss‚1.1704241054899551 /loss/FRt1/detection_loss‚0.6323408867921418 1loss/SRqqt1/detection_loss‚1.3692863100794106 /loss/FBt1/detection_loss‚0.9657124045322488 1loss/FBt1/assignment_loss‚0.09412097159710112 1loss/FBt2/assignment_loss‚0.11489823628006472 1loss/SRqqt2/detection_loss‚0.7930621958874982 /loss/FBt2/detection_loss‚1.4406077783399571 trainer/global_step‚899 _runtime‚1573.604485432 .loss/FRt2/assignment_loss‚2.09096744757139^¾ì‚ 2loss/SRqqt2/assignment_loss‚1.1687392676925015 /loss/FRt2/detection_loss‚1.4413966486035295 &loss/total_loss‚14.736564866916158 epoch‚2 % _timestamp‚1.7843314130192788e+09 _step‚73 0loss/FRt1/assignment_loss‚3.4550086140503615 2loss/SRqqt1/assignment_loss‚1.1704241054899551 /loss/FRt1/detection_loss‚0.6323408867921418 1loss/SRqqt1/detection_loss‚1.3692863100794106 /loss/FBt1/detection_loss‚0.9657124045322488 1loss/FBt1/assignment_loss‚0.09412097159710112 1loss/FBt2/assignment_loss‚0.11489823628006472 1loss/SRqqt2/detection_loss‚0.7930621958874982 /loss/FBt2/detection_loss‚1.4406077783399571 trainer/global_step‚899 _runtime‚1573.604485432 .loss/FRt2/assignment_loss‚2.09096744757139Iß —Ÿí‚(:” –ùêÒæ«ÂÈ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.838' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.967600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì¨Ûîî‚(:ã –ùêÒøŠÊ disk.dm-1.in‚0 disk.sda.out‚ 42.00390625" disk.dm-0.out‚9096.25146484375 disk.sda2.in‚0! proc.memory.rssMB‚ 2837.703125 disk.sda1.in‚0 disk.sda3.out‚ 42.00390625 disk.rbd2.in‚0* proc.memory.percent‚17.319965362548828 proc.cpu.threads‚17 network.sent‚ 1.562166e+06 disk.sda.in‚0! disk.nvme0n1.in‚ 2045.00390625% disk.nvme0n1.out‚9096.25146484375 disk.sda3.in‚0$ disk./.usageGB‚583.7969703674316 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 2045.00390625 disk.sda1.out‚0' disk.nvme0c0n1.out‚9096.25146484375 disk.dm-1.out‚ 42.00390625 disk.rbd1.out‚4.296875 disk.rbd0.out‚0# memory_percent‚88.1140947341919 disk.drbd2165.out‚0 cpu‚25.47035079554593 disk.dm-0.in‚ 2045.00390625 disk.rbd0.in‚0 disk.rbd2.out‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1947.38671875 disk.drbd2165.in‚0 network.recv‚430391* disk./.usagePercent‚39.179104519643026«ö¥”ï‚(:‰ ¥ùêÒå¾ÏÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚270.654' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0߉º9ãð‚(:Ø ¥ùêÒºæõÉ disk.dm-1.in‚0 disk.rbd1.out‚4.296875$ memory_percent‚88.11585903167725) disk./.usagePercent‚39.17909811945467 disk.sda1.out‚0 disk.rbd2.in‚0 disk.dm-1.out‚ 42.421875 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.625215e+06 disk./.usageGB‚ 583.796875# disk.nvme0c0n1.in‚ 2106.92578125" disk.dm-0.out‚9106.85009765625 disk.sda.in‚0 network.recv‚434957! disk.nvme0n1.in‚ 2106.92578125 disk.sda3.out‚ 42.421875 disk.drbd2165.in‚0 disk.drbd2165.out‚0) proc.memory.percent‚17.32003688812256 cpu‚25.469393153404713 disk.sda1.in‚0 disk.sda3.in‚0) proc.memory.availableMB‚ 1947.09765625' disk.nvme0c0n1.out‚9106.85009765625 disk.rbd0.in‚0 disk.rbd2.out‚0 disk.sda2.out‚0 disk.sda.out‚ 42.421875% disk.nvme0n1.out‚9106.85009765625# proc.memory.rssMB‚ 2837.71484375 disk.dm-0.in‚ 2106.92578125 disk.rbd0.out‚0 disk.sda2.in‚0X羟ñ‚(:” ´ùêÒ¯À«È gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚280.079' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.015800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èûä£èò‚(:Ý ´ùêÒøŠ€Ê disk.dm-1.out‚ 42.6484375 disk.sda.out‚ 42.6484375 disk.sda3.out‚ 42.6484375# memory_percent‚88.1173849105835# proc.memory.rssMB‚ 2838.51171875 cpu‚25.468598605479293 disk.rbd1.out‚4.359375! disk.dm-0.out‚9114.9892578125 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0$ disk.nvme0n1.out‚9114.9892578125 network.recv‚438172 disk.dm-0.in‚ 2106.92578125! disk.nvme0n1.in‚ 2106.92578125 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd2165.in‚0 network.sent‚ 1.63037e+06 disk.drbd2165.out‚0 disk.rbd2.out‚0) disk./.usagePercent‚39.17909658340947$ disk./.usageGB‚583.7968521118164 disk.rbd1.in‚0 disk.sda.in‚0 disk.rbd2.in‚0) proc.memory.availableMB‚ 1946.84765625) proc.memory.percent‚17.32490062713623# disk.nvme0c0n1.in‚ 2106.92578125& disk.nvme0c0n1.out‚9114.9892578125 proc.cpu.threads‚17»KŸó‚(:” ÃùêÒﶰÇ gpu.0.gpu‚79.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚258.814' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.762800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^´?kÈô‚(:½ ÃùêÒ¾ûÎÉ$ memory_percent‚94.04964447021484% proc.memory.availableMB‚ 974.90625# proc.memory.rssMB‚ 2838.51171875) proc.memory.percent‚17.32490062713623 disk.rbd0.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.sda3.out‚43 disk.sda.in‚0" disk.nvme0c0n1.in‚ 2107.7578125 disk.sda1.in‚0 disk.rbd2.out‚0 disk.sda2.in‚0 disk.dm-0.out‚9125.005859375 disk.nvme0n1.in‚ 2107.7578125% disk.nvme0c0n1.out‚9125.005859375 cpu‚25.46798554894882 disk.rbd2.in‚0 disk.sda.out‚43 disk.drbd2165.out‚0 disk.sda3.in‚0 network.sent‚ 1.635353e+06$ disk./.usageGB‚583.7966232299805 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚43 disk.rbd1.out‚4.359375# disk.nvme0n1.out‚9125.005859375 disk.sda1.out‚0 network.recv‚441439* disk./.usagePercent‚39.179081222957414 disk.dm-0.in‚ 2107.7578125‹0h”õ‚(:‰ ÒùêÒâÖËÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚271.984' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÅÁ|Üö‚(:Ñ ÒùêÒÏžöÉ disk.sda1.out‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 974.453125) disk./.usagePercent‚39.17908327101768 disk.dm-0.in‚ 2108.28125 disk.rbd0.in‚0 disk.sda.out‚ 43.3046875 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚9132.3955078125 disk.sda1.in‚0) proc.memory.percent‚17.32490062713623 cpu‚25.467361452951284! disk.dm-0.out‚9132.3955078125 disk.sda3.out‚ 43.3046875 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 43.3046875$ memory_percent‚94.05241012573242 proc.cpu.threads‚17 disk.sda3.in‚0 network.sent‚ 1.640238e+06# proc.memory.rssMB‚ 2838.51171875 disk.rbd2.out‚0 disk.drbd2165.in‚0 disk.sda.in‚0 network.recv‚444665$ disk./.usageGB‚583.7966537475586 disk.nvme0c0n1.in‚ 2108.28125 disk.nvme0n1.in‚ 2108.28125 disk.rbd0.out‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚9132.3955078125 disk.rbd2.in‚0 disk.rbd1.out‚4.4375®—”÷‚(:‰ áùêÒ׆ÂÇ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚289.166' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.8332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À]SÚø‚(:Ï áùêÒ˾´É) proc.memory.percent‚17.32490062713623 network.sent‚ 1.645173e+06! disk.nvme0n1.out‚ 9140.8359375 disk.sda3.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 43.6796875" disk.nvme0c0n1.in‚ 2196.2421875# disk.nvme0c0n1.out‚ 9140.8359375 cpu‚25.46647626011706 disk.rbd1.out‚4.4375 disk.drbd2165.out‚0( proc.memory.availableMB‚ 974.69921875) disk./.usagePercent‚39.17907917489713$ disk./.usageGB‚583.7965927124023 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.sda2.in‚0 disk.dm-0.out‚ 9140.8359375 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 network.recv‚447543 disk.dm-1.in‚0 disk.sda.out‚ 43.6796875# proc.memory.rssMB‚ 2838.51171875 disk.sda.in‚0 disk.drbd2165.in‚0$ memory_percent‚94.05090808868408 proc.cpu.threads‚17 disk.nvme0n1.in‚ 2196.2421875 disk.sda1.in‚0 disk.sda3.out‚ 43.6796875 disk.rbd0.in‚0 disk.dm-0.in‚ 2196.2421875B‘_2ù- _wandbruntime‚1655  _runtime‚1655LˆFÇúÁ ' lr-AdamW/pg1‚0.00048154000138389834 ' lr-AdamW/pg2‚0.00048154000138389834 trainer/global_step‚949 % _timestamp‚1.7843314950641484e+09 _runtime‚1655.649280217 _step‚74þ|‚ËûÅ _runtime‚1655.649280217 _step‚74 ' lr-AdamW/pg1‚0.00048154000138389834 ' lr-AdamW/pg2‚0.00048154000138389834 trainer/global_step‚949 % _timestamp‚1.7843314950641484e+09J“Û:€2ü- _wandbruntime‚1655  _runtime‚1655n9v¥ýÓO H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 890983026176 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16267611 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2692282 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚272616823756093 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚271844158717952 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2437890 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271726020964352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8064083 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73900431241216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚1655.650927365 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚75048278898688 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚272615453727807 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚272735115570688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271725995536384 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 890802791741 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 889458191423 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75047876270592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73900063150080 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16277589 WKDeviceStatsMonitor.on_train_batc<çˆ4h_start/inactive_split.small_pool.allocated‚2692524 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _step‚75 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272735115570688 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1147813120512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5130414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272736487133184 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271844132544512 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8064077 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8064083 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚271844132544512 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 892328415232 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 890983026176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1147847657472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224µÄuÝ'þ×O SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2692282 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚272616823756093 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚271844158717952 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2437890 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271726020964352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8064083 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73900431241216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚1655.650927365 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚75048278898688 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚272615453727807 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚272735115570688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271725995536384 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 890802791741 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 889458191423 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75047876270592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73900063150080 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16277589 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2692524 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _step‚75 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272735115570688 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1147813120512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5130414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272736487133184 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271844132544512 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8064077 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8064083 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚271844132544512 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 892328415232 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 890983026176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1147847657472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 890983026176 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16267611 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242KG»cî2ÿ- _wandbruntime‚1657  _runtime‚1657Ý%ã&€ÝM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 890381657669 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5135656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 893253457920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 891908061696 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5135420 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2440433 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2440428 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚272003877314560 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _runtime‚1657.074939241 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16284295 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8221998 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73977549557760 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 891726258099 % _timestamp‚1.7843314964893928e+09 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _step‚76 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8072283 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚273014041336320 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8212012 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚75126974080000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1149056431104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16284295 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚272003902742528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚273015412905984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8072289 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272122133274624 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8221998 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8212012 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273015412905984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2695223 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚272895629000627 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 891908061696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚272894258972229 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16294287 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚75126571459072 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 893253457920 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16294287 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0·xq·áM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚272122133274624 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73977917648896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5135656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 893253457920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 891908061696 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5135420 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2440433 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2440428 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚272003877314560 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _runtime‚1657.074939241 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16284295 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8221998 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73977549557760 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 891726258099 % _timestamp‚1.7843314964893928e+09 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _step‚76 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8072283 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚273014041336320 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8212012 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚75126974080000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1149056431104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16284295 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚272003902742528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚273015412905984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8072289 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272122133274624 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8221998 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8212012 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273015412905984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2695223 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚272895629000627 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 891908061696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚272894258972229 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16294287 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytesÍL/0.all.freed‚75126571459072 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 893253457920 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16294287 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 890381657669 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286681468006912 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 938644345856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8629286 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚285620084445184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 trainer/global_step‚999 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286555779749715 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7843315787293792e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚285744169050112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 937298956800 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 938644345856 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚285744195223552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2565035 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8474377 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8639258 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286557149778001 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285620109873152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚285744169050112 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2830179 _runtime‚1739.317184627DœiÊÝ'”×O PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5395219 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8474383 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286681468006912 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 938644345856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8629286 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚285620084445184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 trainer/global_step‚999 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286555779749715 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7843315787293792e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚285744169050112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 937298956800 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 938644345856 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚285744195223552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2565035 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8474377 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8639258 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286557149778001 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285620109873152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚285744169050112 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2830179 _runtime‚1739.317184627 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2565040 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2829934 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8474383 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17113641 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8629286 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1207998608384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1207964071424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 937039904849 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 938223992320 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5400256 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 939569388544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529792 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286834584994137 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚286022195953664 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 936618770777 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17120347 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17120347 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 % _timestamp‚1.7843315801527457e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 939569388544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1209201623040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2567578 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2832917 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286022169780224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚286022169780224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚286961765342208 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2567583 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2832678 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79061037035008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17130339 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8637764 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚286022195953664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8482583 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8482583 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚286961765342208 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8637764 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286960393772544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8647750 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17130339 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8482589 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640˜³ªmæ&™àM QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8647750 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17130339 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8482589 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8647750 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79061439655936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1209167093248 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 938223992320 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5400256 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 939569388544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529792 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286834584994137 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚286022195953664 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 936618770777 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17120347 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17120347 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 % _timestamp‚1.7843315801527457e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 939569388544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1209201623040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2567578 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2832917 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286022169780224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚286022169780224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚286961765342208 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2567583 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2832678 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79061037035008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17130339 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8637764 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚286022195953664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8482583 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8482583 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚286961765342208 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8637764 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286960393772544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008PúŽJ½2š- _wandbruntime‚1740  _runtime‚1740‰q)Q›û % _timestamp‚1.7843315802271495e+09 _step‚81 0loss/FBt2/assignment_loss‚0.1947425277993164 &loss/total_loss‚14.836156541653581 trainer/global_step‚999 /loss/FRt1/assignment_loss‚3.274929371373407 2loss/SRqqt2/assignment_loss‚1.1360048321052911 /loss/FBt2/detection_loss‚1.6851040492100375 epoch‚2 _runtime‚1740.812271676 1loss/FBt1/assignment_loss‚0.14532270423904678 .loss/FRt2/detection_loss‚1.384559087991914 1loss/SRqqt2/detection_loss‚0.7817261294443526 .loss/FBt1/detection_loss‚0.713302628824897 0loss/FRt2/assignment_loss‚2.1574198036537298 2loss/SRqqt1/assignment_loss‚1.2692852313676388 .loss/FRt1/detection_loss‚0.678674080542482 1loss/SRqqt1/detection_loss‚1.41508609510146748Ï^Ôœÿ .loss/FRt1/detection_loss‚0.678674080542482 1loss/SRqqt1/detection_loss‚1.4150860951014674 % _timestamp‚1.7843315802271495e+09 _step‚81 0loss/FBt2/assignment_loss‚0.1947425277993164 &loss/total_loss‚14.836156541653581 trainer/global_step‚999 /loss/FRt1/assignment_loss‚3.274929371373407 2loss/SRqqt2/assignment_loss‚1.1360048321052911 /loss/FBt2/detection_loss‚1.6851040492100375 epoch‚2 _runtime‚1740.812271676 1loss/FBt1/assignment_loss‚0.14532270423904678 .loss/FRt2/detection_loss‚1.384559087991914 1loss/SRqqt2/detection_loss‚0.7817261294443526 .loss/FBt1/detection_loss‚0.713302628824897 0loss/FRt2/assignment_loss‚2.1574198036537298 2loss/SRqqt1/assignment_loss‚1.2692852313676388Q.)Ÿ‚(:” ÊúêÒµÑ›È gpu.0.gpu‚97.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚191.777' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚38.355399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚œ$¸ž‚(:­ ÊúêÒÛºüÉ disk.sda2.out‚0 network.sent‚ 1.784869e+06 disk.rbd1.in‚0$ disk.nvme0c0n1.out‚ 9202.95703125 disk.dm-0.out‚ 9202.95703125( proc.memory.availableMB‚ 974.20703125 disk.nvme0n1.in‚ 2268.5390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 47.89453125 disk.sda2.in‚0 disk.dm-1.in‚0 network.recv‚473601 disk.dm-1.out‚ 47.89453125# proc.memory.rssMB‚ 2838.51171875) disk./.usagePercent‚39.14356913784527" disk.nvme0c0n1.in‚ 2268.5390625 disk.dm-0.in‚ 2268.5390625$ memory_percent‚94.05391216278076$ disk./.usageGB‚583.2674674987793 disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 47.89453125 cpu‚25.46043215349954" disk.nvme0n1.out‚ 9202.95703125 disk.rbd1.out‚4.78125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0) proc.memory.percent‚17.32490062713623é§Ó}”Ÿ‚(:‰ ÙúêÒìÕîÈ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚233.284' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.6568 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y-Df´ ‚(:© ÙúêÒü£³É$ disk./.usageGB‚583.2679977416992 disk.dm-0.in‚2357.875' disk.nvme0c0n1.out‚9205.87646484375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0* disk./.usagePercent‚39.143604722892526 disk.sda.in‚0 disk.drbd2165.out‚0 cpu‚25.459526650343165 network.recv‚478165 disk.rbd0.out‚0 network.sent‚ 1.837227e+06 disk.sda1.in‚0 disk.rbd1.out‚4.78125 disk.sda.out‚ 48.33203125$ memory_percent‚94.05627250671387! proc.memory.rssMB‚ 2838.515625 proc.cpu.threads‚17 disk.nvme0c0n1.in‚2357.875 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚9205.87646484375 disk.dm-1.out‚ 48.33203125' proc.memory.availableMB‚ 973.8203125) proc.memory.percent‚17.32492446899414 disk.nvme0n1.in‚2357.875 disk.sda3.out‚ 48.33203125" disk.dm-0.out‚9205.87646484375)àÛ┡‚(:‰ èúêÒ”–éÈ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚241.613' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.3226 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥ìóྡྷ‚(:³ èúêÒÈòÙÉ network.recv‚481194 disk.dm-1.out‚ 49.15234375 disk.sda.out‚ 49.15234375" disk.nvme0c0n1.in‚ 2358.8515625% disk.nvme0n1.out‚9225.35400390625 disk.sda3.in‚0' proc.memory.availableMB‚ 973.5859375 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.drbd2165.out‚0$ memory_percent‚94.05770301818848 network.sent‚ 1.842284e+06) proc.memory.percent‚17.32492446899414 disk.rbd1.out‚4.84375 disk.sda3.out‚ 49.15234375 disk.rbd0.in‚0 cpu‚25.45874784865688 disk.sda2.out‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚9225.35400390625 disk.dm-0.in‚ 2358.8515625* disk./.usagePercent‚39.013931018626536 disk.sda2.in‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2838.515625# disk./.usageGB‚581.335765838623" disk.dm-0.out‚9225.35400390625 disk.nvme0n1.in‚ 2358.8515625 disk.sda1.in‚0 disk.sda1.out‚0=™ŸŸ£‚(:” ÷úêÒ¸™óÇ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚288.151' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.630199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»÷úH²¤‚(:§ ÷úêÒþ¦É) proc.memory.percent‚17.32492446899414% disk.nvme0n1.out‚9229.64306640625 disk.nvme0n1.in‚ 2358.8515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 2358.8515625 disk.dm-0.in‚ 2358.8515625 proc.cpu.threads‚17$ memory_percent‚94.05746459960938$ disk./.usageGB‚581.3360786437988 disk.sda2.out‚0 disk.dm-1.out‚49.53125' disk.nvme0c0n1.out‚9229.64306640625 network.sent‚ 1.847135e+06) disk./.usagePercent‚39.01395201124434 disk.rbd0.out‚0! proc.memory.rssMB‚ 2838.515625 cpu‚25.458002885409506 disk.sda3.in‚0" disk.dm-0.out‚9229.64306640625 network.recv‚484868 disk.sda3.out‚49.53125 disk.dm-1.in‚0 disk.rbd1.out‚4.84375 disk.sda.in‚0 disk.drbd2165.out‚0# proc.memory.availableMB‚973.625 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda.out‚49.53125ÿòõ°”¥‚(:‰ †ûêÒáꪂ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚259.878' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.9756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MËKPĦ‚(:¹ †ûêÒÌýáÈ disk.sda1.in‚0$ memory_percent‚94.05906200408936( proc.memory.availableMB‚ 973.36328125) proc.memory.percent‚17.32492446899414 cpu‚25.457273887949338 disk.sda3.in‚0 network.sent‚ 1.852071e+06 network.recv‚488036 disk.dm-1.in‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 2359.68359375 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚9241.10107421875 disk.dm-0.in‚ 2359.68359375 disk.sda3.out‚ 49.83203125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0) disk./.usagePercent‚39.01396097150804$ disk./.usageGB‚581.3362121582031 disk.rbd1.out‚4.921875 disk.sda2.in‚0 disk.rbd1.in‚0" disk.dm-0.out‚9241.10107421875% disk.nvme0n1.out‚9241.10107421875! proc.memory.rssMB‚ 2838.515625 disk.dm-1.out‚ 49.83203125 disk.sda.out‚ 49.83203125 disk.sda2.out‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 2359.68359375#òÖâ2§- _wandbruntime‚1822  _runtime‚1822D-ÎcȨ _runtime‚1822.631865772 _step‚82 ' lr-AdamW/pg1‚0.00048116933285163134 ' lr-AdamW/pg2‚0.00048116933285163134 trainer/global_step‚1049 % _timestamp‚1.7843316620467043e+09d65*Ì©Æ trainer/global_step‚1049 % _timestamp‚1.7843316620467043e+09 _runtime‚1822.631865772 _step‚82 ' lr-AdamW/pg1‚0.00048116933285163134 ' lr-AdamW/pg2‚0.00048116933285163134R¡šœ2ª- _wandbruntime‚1822  _runtime‚1822ÖŒòˆû «ÔO K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8884677 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1268111321088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17939715 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 984960276480 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 984960276480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚299514198781952 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8884683 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9055038 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2692190 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81649072009216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _step‚83 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8884683 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9055038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2967369 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82916780702208 YHDeviceSt™h©ßatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 981932417639 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300629192005632 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299644231729152 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 983614887424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚1049 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 983614887424 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9065010 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8884677 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5659802 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299644205555712 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17949693 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9065010 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82917183330304 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300627820443136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚300496105771623 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7843316620480404e+09 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1268076784128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚299514173353984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300627820443136 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81648703918080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300627820443136 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81648703918080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17939715 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 984960276480 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 984960276480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚299514198781952 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8884683 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9055038 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2692190 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81649072009216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _step‚83 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8884683 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9055038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2967369 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82916780702208 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 981932417639 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300629192005632 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299644231729152 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 983614887424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚1049 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 983614887424 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9065010 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8884677 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5659802 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299644205555712 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17949693 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9065010 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82917183330304 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300627820443136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚300496105771623 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7843316620480404e+09 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1268076784128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚299514173353984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Sºÿ°ë2­- _wandbruntime‚1824  _runtime‚1824¨ó…àä&®ÞM PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9063516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 985885319168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 984539922944 % _timestamp‚1.7843316634747741e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17956399 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8892883 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9063516 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚299792080560128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 984200484315 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8892883 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 985885319168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17966391 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 trainer/global_step‚1049 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2694733 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299922206285824 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5664828 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82995873456128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1269280509440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚300776281044443 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8892889 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚300774911016045 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5665070 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚1824.060610145 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚299922206285824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17966391 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 trainer/global_step‚1049 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2694733 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299922206285824 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5664828 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82995873456128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1269280509440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚300776281044443 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8892889 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚300774911016045 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5665070 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚1824.060610145 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚299922206285824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9063516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 985885319168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_f|‡Þtrain_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 984539922944 % _timestamp‚1.7843316634747741e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17956399 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8892883 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9063516 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚299792080560128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 984200484315 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8892883 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 985885319168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 Bå÷à'ÃÚO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1029930818048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18785745 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9490762 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314574172879360 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5924381 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86851628718080 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9294977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3104785 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313544268234752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1031276207104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1029930818048 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3105041 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚314436431793531 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9294983 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9480790 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2819340 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313544242061312 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 % _timestamp‚1.7843317452930598e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86851226089984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9294977 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313544242061312 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1328201787904 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9480790 _step‚87 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚314575544441856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 trainer/global_step‚1099 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1028169530747 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313544268234752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85523392393216 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2819335 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85523024302080 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9294983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008±­µÄÞO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1328236324864 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313408262262784 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9294983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1029930818048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18785745 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9490762 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314574172879360 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5924381 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86851628718080 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9294977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3104785 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313544268234752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1031276207104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1029930818048 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3105041 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚314436431793531 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9294983 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9480790 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2819340 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313544242061312 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 % _timestamp‚1.7843317452930598e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86851226089984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9294977 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313544242061312 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1328201787904 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9480790 _step‚87 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚314575544441856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 trainer/global_step‚1099 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1028169530747 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚;žO/ 9978 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313544268234752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85523392393216 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2819335 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85523024302080 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9303183 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚1099 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚313822268964864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9303189 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313822242791424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18792451 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9489268 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85600878800896 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18802443 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚313822268964864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1032201249792 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9303183 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9499254 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18802443 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9499254 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5929402 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5929658 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1329428689920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚88 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1907.289729624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1030855853568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1030437597423 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18792451 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1032201249792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2821883 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3107524 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85600510709760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1329392062976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995Váî&ÇèM _step‚88 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1907.289729624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1030855853568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1030437597423 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18792451 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1032201249792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2821883 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3107524 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85600510709760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1329392062976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313686169468928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9303183 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚1099 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚313822268964864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9303189 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313822242791424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18792451 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9489268 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85600878800896 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18802443 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚313822268964864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1032201249792 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9303183 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9499254 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18802443 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9499254 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5929402 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5929658 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1329428689920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0XÒÒÁ'2È- _wandbruntime‚1907  _runtime‚1907ä'š¤É€ 0loss/FRt1/assignment_loss‚3.1175288081298844 1loss/FBt2/assignment_loss‚0.16629242829419008 epoch‚2 trainer/global_step‚1099 0loss/FRt2/assignment_loss‚2.0973152460278692 1loss/SRqqt2/detection_loss‚0.7683466230292955 _step‚89 2loss/SRqqt1/assignment_loss‚1.1456567558947772 /loss/FRt1/detection_loss‚0.6357664170298625 /loss/FBt1/detection_loss‚0.8157361015283084 _runtime‚1907.362293407 2loss/SRqqt2/assignment_loss‚1.0478593328925503 1loss/FBt1/assignment_loss‚0.10726189216720348 /loss/FRt2/detection_loss‚1.4244183052809456 1loss/SRqqt1/detection_loss‚1.3906926304811522 /loss/FBt2/detection_loss‚1.3585002221937836 %loss/total_loss‚14.07537476294982 % _timestamp‚1.7843317467771606e+09öáb Ê„ 2loss/SRqqt1/assignment_loss‚1.1456567558947772 /loss/FRt1/detection_loss‚0.6357664170298625 /loss/FBt1/detection_loss‚0.8157361015283084 _runtime‚1907.362293407 2loss/SRqqt2/assignment_loss‚1.0478593328925503 1loss/FBt1/assignment_loss‚0.10726189216720348 /loss/FRt2/detection_loss‚1.4244183052809456 1loss/SRqqt1/detection_loss‚1.3906926304811522 /loss/FBt2/detection_loss‚1.3585002221937836 %loss/total_loss‚14.07537476294982 % _timestamp‚1.7843317467771606e+09 0loss/FRt1/assignment_loss‚3.1175288081298844 1loss/FBt2/assignment_loss‚0.16629242829419008 epoch‚2 trainer/global_step‚1099 0loss/FRt2/assignment_loss‚2.0973152460278692 1loss/SRqqt2/detection_loss‚0.7683466230292955 _step‚89Yóêˆ?“Ë‚(:ˆ ïûêÒÓà¨É gpu.0.gpu‚94.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚228.715' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.743 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ya7¢®Ì‚(:£ ïûêÒ©²Ê disk.sda3.out‚ 54.765625 disk.sda.in‚0 disk.dm-1.out‚ 54.765625 proc.cpu.threads‚17 disk.drbd2165.out‚0" disk.dm-0.out‚9303.46142578125' proc.memory.availableMB‚ 974.6484375 network.recv‚513430$ disk./.usageGB‚581.3366851806641 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 2509.6875 cpu‚25.45234318706897 disk.nvme0n1.in‚ 2509.6875% disk.nvme0n1.out‚9303.46142578125$ memory_percent‚94.05121803283691 network.sent‚ 1.945269e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0) disk./.usagePercent‚39.01399271644229 disk.rbd0.in‚0 disk.rbd1.out‚5.1875! proc.memory.rssMB‚ 2838.515625 disk.rbd0.out‚0 disk.sda.out‚ 54.765625 disk.sda3.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 2509.6875' disk.nvme0c0n1.out‚9303.46142578125) proc.memory.percent‚17.32492446899414ÁÙu”Í‚(:‰ þûêÒú­¢Ç gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚259.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"e^'Á΂(:¶ þûêÒëêœÉ$ disk./.usageGB‚581.3366813659668 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 56.0546875) proc.memory.percent‚17.32492446899414 disk.sda2.in‚0 disk.sda1.out‚0" disk.dm-0.out‚9315.40380859375! proc.memory.rssMB‚ 2838.515625 disk.drbd2165.in‚0 disk.dm-0.in‚ 2510.06640625 disk.sda1.in‚0 disk.rbd1.out‚5.265625 disk.sda3.out‚ 56.0546875 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.sda.out‚ 56.0546875# disk.nvme0c0n1.in‚ 2510.06640625' disk.nvme0c0n1.out‚9315.40380859375( proc.memory.availableMB‚ 974.37109375 network.sent‚ 2.010283e+06! disk.nvme0n1.in‚ 2510.06640625$ memory_percent‚94.05291080474854 proc.cpu.threads‚17 network.recv‚517764 disk.rbd1.in‚0 disk.sda.in‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚9315.40380859375 cpu‚25.451587302094264) disk./.usagePercent‚39.01399246043476oBë’Ï‚(:‡ üêÒרÖÇ gpu.0.gpu‚86.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚276.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒÌi¹Ð‚(:® üêÒ¬ÚýÉ disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚9327.47216796875 cpu‚25.45092196578469 disk.sda3.out‚ 58.0546875 disk.sda2.out‚0" disk.dm-0.out‚9327.47216796875' proc.memory.availableMB‚ 974.3671875! proc.memory.rssMB‚ 2838.515625 network.recv‚521457) disk./.usagePercent‚39.01398452420119$ disk./.usageGB‚581.3365631103516 disk.drbd2165.out‚0 network.sent‚ 2.013205e+06 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 58.0546875$ memory_percent‚94.05293464660645 disk.rbd1.out‚5.265625 proc.cpu.threads‚17% disk.nvme0n1.out‚9327.47216796875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 58.0546875! disk.nvme0c0n1.in‚ 2601.109375 disk.dm-0.in‚ 2601.109375 disk.nvme0n1.in‚ 2601.109375) proc.memory.percent‚17.3249244689941460×¹žÑ‚(:“ œüêÒÔ­ÁÇ gpu.0.gpu‚86.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚280.287' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.05739999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#š|•¾Ò‚(:³ œüêÒ¸ê©É$ disk./.usageGB‚581.3363914489746 disk.dm-1.in‚0 disk.dm-1.out‚ 59.17578125 disk.sda3.out‚ 59.17578125$ memory_percent‚94.05443668365479 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 59.17578125! disk.nvme0c0n1.in‚ 2601.109375' disk.nvme0c0n1.out‚9335.61669921875 disk.dm-0.in‚ 2601.109375) proc.memory.percent‚17.32492446899414 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda3.in‚0 network.recv‚524706) disk./.usagePercent‚39.01397300386215 disk.sda2.in‚0 disk.rbd1.out‚5.328125 disk.drbd2165.in‚0% disk.nvme0n1.out‚9335.61669921875 network.sent‚ 2.018172e+06" disk.dm-0.out‚9335.61669921875( proc.memory.availableMB‚ 974.12109375 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 2601.109375! proc.memory.rssMB‚ 2838.515625 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 cpu‚25.450304408788885ìîyŸÓ‚(:” «üêÒÜÁÅÈ gpu.0.gpu‚85.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚291.919' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.383799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æš‡C¼Ô‚(:± «üêÒ÷•Ê* disk./.usagePercent‚39.013976843975165 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0% disk.nvme0c0n1.out‚9342.712890625 disk.sda3.out‚ 59.6796875$ memory_percent‚94.05291080474854 cpu‚25.449782107831304 disk.sda.out‚ 59.6796875# disk.nvme0c0n1.in‚ 2602.07421875! disk.nvme0n1.in‚ 2602.07421875( proc.memory.availableMB‚ 974.37109375 disk.dm-1.out‚ 59.6796875 disk.drbd2165.out‚0 disk.dm-0.out‚9342.712890625 network.recv‚527733$ disk./.usageGB‚581.3364486694336 disk.rbd1.out‚5.328125 disk.sda.in‚0 disk.drbd2165.in‚0# disk.nvme0n1.out‚9342.712890625 disk.sda1.out‚0! proc.memory.rssMB‚ 2838.515625 network.sent‚ 2.022923e+06 disk.sda2.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 2602.07421875) proc.memory.percent‚17.32492446899414 disk.rbd0.out‚0 disk.sda3.in‚0 proc.cpu.threads‚173YÒ2Õ- _wandbruntime‚1989  _runtime‚1989™†Ô˜ÆÖÀ & lr-AdamW/pg1‚0.0004807370236100366 & lr-AdamW/pg2‚0.0004807370236100366 trainer/global_step‚1149 % _timestamp‚1.7843318288503218e+09 _runtime‚1989.435567196 _step‚90yÌxÊ×Ä _runtime‚1989.435567196 _step‚90 & lr-AdamW/pg1‚0.0004807370236100366 & lr-AdamW/pg2‚0.0004807370236100366 trainer/global_step‚1149 % _timestamp‚1.7843318288503218e+09Z±ë¬2Ø- _wandbruntime‚1989  _runtime‚1989¹ú¾e_ÙÚO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9705277 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9705283 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89397712777216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _runtime‚1989.438079079 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328520525315584 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3242037 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚327444278566912 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1077592137728 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚328521896878080 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327444278566912 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚90785557256704 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328520525315584 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚328378127843725 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1075751244173 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9916514 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚328376757815439 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1388247107584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1388212570624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1077592137728 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6188522 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1076246748672 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 % _timestamp‚1.7843318288517745e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9705277 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89397344686080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1076246748672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2946485 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6188766 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚327444304740352 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2946490 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9705283 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚327444304740352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚327302376599552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1074406643855 _step‚91 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚90785959884800 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19611819 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9906542 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9916514 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9705277 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9705283 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89397712777216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _runtime‚1989.438079079 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328520525315584 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3242037 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚327444278566912 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1077592137728 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚328521896878080 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327444278566912 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚90785557256704 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328520525315584 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚328378127843725 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1075751244173 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9916514 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚328376757815439 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1388247107584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1388212570624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1077592137728 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6188522 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1076246748672 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 % _timestamp‚1.7843318288517745e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9705277 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89397344686080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1076246748672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2946485 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6188766 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚327444304740352 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2946490 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9705283 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚327444304740352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚327302376599552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1074406643855 _step‚91 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244[J{-á2Û- _wandbruntime‚1990  _runtime‚1990º,ê&ÜäM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1076674710531 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9915020 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚327580258377728 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9925006 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚327580232949760 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6193806 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚328800822650880 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19638495 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚328655563059861 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚328656933088259 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1078517180416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9925006 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19628503 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327722279297024 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9915020 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1077171784192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚327722279297024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89474831093760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2949028 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 % _timestamp‚1.7843318302789075e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19638495 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9713489 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚327722305470464 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327722305470464 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3244778 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0ˆLá¦\ÝèM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2949028 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 % _timestamp‚1.7843318302789075e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19638495 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9713489 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚327722305470464 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327722305470464 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3244778 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1076674710531 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9915020 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚327580258377728 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9925006 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚327580232949760 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6193806 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚328800822650880 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19638495 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚328655563059861 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚328656933088259 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1078517180416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9925006 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19628503 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327722279297024 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9915020 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1077171784192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚327722279297024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89474831093760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0\Öâ†>2Þ- _wandbruntime‚1990  _runtime‚1990j8¿ÿßù 0loss/FRt2/assignment_loss‚2.0446802855398682 2loss/SRqqt2/assignment_loss‚1.0099143549853307 /loss/FRt2/detection_loss‚1.4595402667915873 /loss/FBt2/detection_loss‚0.9606905460357666 _step‚93 epoch‚2 % _timestamp‚1.7843318303479357e+09 %loss/total_loss‚13.92093586451255 /loss/FRt1/assignment_loss‚3.117043087980161 1loss/SRqqt1/assignment_loss‚1.241246732458703 /loss/FRt1/detection_loss‚0.6328195562563584 1loss/SRqqt2/detection_loss‚0.7921485554474365 trainer/global_step‚1149 _runtime‚ 1990.93307595 1loss/FBt1/assignment_loss‚0.05309079728858073 /loss/FBt2/assignment_loss‚0.388330169494611 1loss/SRqqt1/detection_loss‚1.3734369771142976 -loss/FBt1/detection_loss‚0.84799453511985õc7†àý /loss/FRt2/detection_loss‚1.4595402667915873 /loss/FBt2/detection_loss‚0.9606905460357666 _step‚93 epoch‚2 % _timestamp‚1.7843318303479357e+09 %loss/total_loss‚13.92093586451255 /loss/FRt1/assignment_loss‚3.117043087980161 1loss/SRqqt1/assignment_loss‚1.241246732458703 /loss/FRt1/detection_loss‚0.6328195562563584 1loss/SRqqt2/detection_loss‚0.7921485554474365 trainer/global_step‚1149 _runtime‚ 1990.93307595 1loss/FBt1/assignment_loss‚0.05309079728858073 /loss/FBt2/assignment_loss‚0.388330169494611 1loss/SRqqt1/detection_loss‚1.3734369771142976 -loss/FBt1/detection_loss‚0.84799453511985 0loss/FRt2/assignment_loss‚2.0446802855398682 2loss/SRqqt2/assignment_loss‚1.0099143549853307]ÀVk¯žá‚(:“ ºüêÒò°³È gpu.0.gpu‚87.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚289.743' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.94859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖTÝ­â‚(:¢ ºüêÒØ׌Ê disk.dm-1.out‚ 59.9921875 network.recv‚530948 disk.drbd2165.in‚0 disk.dm-0.out‚ 9353.18359375 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.449165790681757 network.sent‚ 2.027115e+06 disk.nvme0c0n1.in‚ 2602.90625 disk.dm-0.in‚ 2602.90625" disk.nvme0n1.out‚ 9353.18359375) proc.memory.percent‚17.32492446899414 disk.sda.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 2602.90625 disk.sda3.out‚ 59.9921875 disk.rbd0.out‚0 disk.sda.out‚ 59.9921875$ disk.nvme0c0n1.out‚ 9353.18359375) disk./.usagePercent‚39.01398196412585$ disk./.usageGB‚581.3365249633789$ memory_percent‚94.05455589294434' proc.memory.availableMB‚ 974.1015625! proc.memory.rssMB‚ 2838.515625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚5.40625Êõ1‚”ã‚(:‰ ÉüêÒŠ¨ûÇ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚321.409' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.2818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OÆѶä‚(:« ÉüêÒ¤öêÉ disk.drbd2165.out‚0 disk.rbd0.out‚0$ memory_percent‚94.05484199523926 disk.rbd1.in‚0 disk.dm-0.in‚ 2694.10546875 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.090033e+06 disk.rbd1.out‚5.40625 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0! proc.memory.rssMB‚ 2838.515625 network.recv‚534951# disk.nvme0c0n1.in‚ 2694.10546875' proc.memory.availableMB‚ 974.0546875) proc.memory.percent‚17.32492446899414) disk./.usagePercent‚39.01398222013338 disk.sda1.out‚0 disk.sda3.out‚60.34375 disk.dm-1.out‚60.34375$ disk./.usageGB‚581.3365287780762 disk.drbd2165.in‚0! disk.dm-0.out‚9358.0849609375! disk.nvme0n1.in‚ 2694.10546875 disk.sda2.in‚0 cpu‚25.448564592935284& disk.nvme0c0n1.out‚9358.0849609375$ disk.nvme0n1.out‚9358.0849609375 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda.out‚60.343757A”žå‚(:“ ØüêÒ—âÇÇ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚336.465' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.29299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zöÐ7¿æ‚(:´ ØüêÒš§”É disk.sda3.out‚ 60.56640625 disk.dm-1.in‚0 disk.dm-1.out‚ 60.56640625 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 2695.0703125' proc.memory.availableMB‚ 974.1796875 disk.rbd0.in‚0$ memory_percent‚94.05407905578613 disk.nvme0n1.in‚ 2695.0703125 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.out‚ 60.56640625 disk.dm-0.in‚ 2695.0703125! proc.memory.rssMB‚ 2838.515625% disk.nvme0n1.out‚9367.87353515625 disk.sda.in‚0) proc.memory.percent‚17.32492446899414 cpu‚25.447860914597335 network.recv‚538277 disk.rbd1.out‚5.53125 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.dm-0.out‚9367.87353515625 network.sent‚ 2.094998e+06) disk./.usagePercent‚39.01400244472859$ disk./.usageGB‚581.3368301391602 disk.sda3.in‚0' disk.nvme0c0n1.out‚9367.87353515625 disk.sda1.out‚0ñüeñ’ç‚(:‡ çüêÒüŽ¿Ç gpu.0.gpu‚6.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚346.282' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.2564 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”ñÏ9¾è‚(:³ çüêÒþ®ÉÉ network.sent‚ 2.099919e+06' disk.nvme0c0n1.out‚9375.23779296875" disk.dm-0.out‚9375.23779296875% disk.nvme0n1.out‚9375.23779296875 disk.rbd0.in‚0$ proc.memory.availableMB‚974.1875( disk./.usagePercent‚39.0139980926005! proc.memory.rssMB‚ 2838.515625 network.recv‚541699 disk.sda3.out‚ 60.86328125 disk.sda2.out‚0 disk.dm-0.in‚ 2695.90234375 disk.dm-1.out‚ 60.86328125 disk.sda.out‚ 60.86328125 disk.drbd2165.in‚0 cpu‚25.447260772200163$ disk./.usageGB‚581.3367652893066! disk.nvme0n1.in‚ 2695.90234375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd2165.out‚0$ memory_percent‚94.05403137207031) proc.memory.percent‚17.32492446899414 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda.in‚0 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 2695.90234375 disk.rbd1.out‚5.53125|‰ÊØ’é‚(:‡ öüêÒœÌÄÈ gpu.0.gpu‚6.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚335.794' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.1588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åútº±ê‚(:¦ öüêÒò΄Ê disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 60.99609375 cpu‚25.44678763127504# disk.nvme0c0n1.in‚ 2695.90234375 disk.dm-0.in‚ 2695.90234375 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚5.609375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0$ memory_percent‚94.05481815338135) disk./.usagePercent‚39.01400782088681 disk.nvme0n1.out‚ 9379.46875) proc.memory.percent‚17.32492446899414 proc.cpu.threads‚17 network.sent‚ 2.105069e+06$ disk./.usageGB‚581.3369102478027 disk.sda3.out‚ 60.99609375 disk.dm-1.out‚ 60.99609375! proc.memory.rssMB‚ 2838.515625 disk.dm-1.in‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 2695.90234375 disk.rbd0.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 974.05859375 network.recv‚544765! disk.nvme0c0n1.out‚ 9379.46875 disk.dm-0.out‚ 9379.46875 disk.sda1.out‚0n’BY’ë‚(:‡ …ýêÒ½®»Ç gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚327.587' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.5174 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01/MA¶ì‚(:« …ýêÒ¨µ–É disk.rbd1.out‚5.609375 cpu‚25.446116640530715 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 61.453125' disk.nvme0c0n1.out‚9387.77978515625 disk.nvme0n1.in‚ 2764.671875) proc.memory.percent‚17.32492446899414 network.sent‚ 2.109929e+06) disk./.usagePercent‚39.01401012495461% disk.nvme0n1.out‚9387.77978515625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚548481 disk.dm-0.in‚ 2764.671875" disk.dm-0.out‚9387.77978515625 disk.sda3.in‚0# memory_percent‚94.0549373626709 proc.cpu.threads‚17 disk.sda2.out‚0$ disk./.usageGB‚581.3369445800781 disk.dm-1.out‚ 61.453125 disk.sda.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 61.453125' proc.memory.availableMB‚ 974.0390625! proc.memory.rssMB‚ 2838.515625 disk.dm-1.in‚0 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 2764.671875 disk.rbd0.in‚0¶%¨¸2í- _wandbruntime‚2073  _runtime‚20731Y)ÆîÀ & lr-AdamW/pg1‚0.0004804977878128604 & lr-AdamW/pg2‚0.0004804977878128604 trainer/global_step‚1199 % _timestamp‚1.7843319125925765e+09 _runtime‚2073.177859634 _step‚94-ÐÎúÊïÄ & lr-AdamW/pg1‚0.0004804977878128604 & lr-AdamW/pg2‚0.0004804977878128604 trainer/global_step‚1199 % _timestamp‚1.7843319125925765e+09 _runtime‚2073.177859634 _step‚94^I [^2ð- _wandbruntime‚2073  _runtime‚2073Û£›>æ'ñàO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341344315072512 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20457849 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚341196465508352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10115577 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1123908068352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10342266 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚95 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10332294 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚94720454391808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20447871 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93271665070080 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1122562679296 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚342466877751808 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3073640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚1199 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 _runtime‚2073.179728055 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1122562679296 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚342317083837347 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚342468249314304 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6453358 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3379718 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚341344341245952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10115583 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3379489 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93272033161216 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚342466877751808 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1448421230592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1448388790784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6453124 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10115583 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10115577 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10332294 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚341196440080384 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚341344315072512 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3073635 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚94720053860864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0…€W.ùòäO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6453124 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10115583 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10115577 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10332294 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚341196440080384 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚341344315072512 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3073635 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚94720053860864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341344315072512 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20457849 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚341196465508352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10115577 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1123908068352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10342266 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚95 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10332294 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚94720454391808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20447871 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93271665070080 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WEDeviceStatsMonitor.on_train_batch_s¢¬¢éñtart/active_bytes.small_pool.freed‚ 1122562679296 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚342466877751808 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3073640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚1199 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 _runtime‚2073.179728055 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1122562679296 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚342317083837347 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚342468249314304 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6453358 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3379718 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚341344341245952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10115583 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3379489 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93272033161216 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚342466877751808 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1448421230592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1448388790784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008_ Iâk2ó- _wandbruntime‚2074  _runtime‚2074åÃïÅñ&ôëM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1124833111040 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3076178 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚341622341976064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10123789 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚341474347286528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6458601 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20464555 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341622341976064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1121567223209 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3076183 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20474547 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10350758 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341474321858560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20474547 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚342597259110167 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93349519568896 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10123783 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10350758 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 _runtime‚2074.602985303 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10123783 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚342745803517440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 $ _timestamp‚1.784331914017501e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6458375 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1123487714816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93349151477760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _step‚96 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1449635664896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10340772 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3382197 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1449603232256 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1122911823639 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1123487714816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚342595889081769 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341622315802624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341622315802624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚221 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚94798754710016 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚1199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚342747175087104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0O…¢ïõ&õïM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10340772 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3382197 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1449603232256 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1122911823639 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1123487714816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚342595889081769 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341622315802624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341622315802624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚221 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚94798754710016 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚1199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚342747175087104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1124833111040 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3076178 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚341622341976064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10123789 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚341474347286528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6458601 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20464555 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341622341976064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1121567223209 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3076183 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20474547 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10350758 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341474321858560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20474547 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚342597259110167 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93349519568896 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10123783 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10350758 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 _runtime‚2074.602985303 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10123783 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚342745803517440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 $ _timestamp‚1.784331914017501e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6458375 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1123487714816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93349151477760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _step‚96 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1449635664896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648`‘ÐI´2ö- _wandbruntime‚2074  _runtime‚20742ª÷ÿ 0loss/FRt2/assignment_loss‚2.1587146031856537 1loss/FBt1/assignment_loss‚0.11382521776747305 &loss/total_loss‚12.850717952633314 trainer/global_step‚1199 1loss/SRqqt1/assignment_loss‚1.142186361404745 2loss/SRqqt2/assignment_loss‚1.0117973919976138 /loss/FRt2/detection_loss‚1.4484863645549524 1loss/SRqqt1/detection_loss‚1.1717267866595096 /loss/FBt2/detection_loss‚0.8285874621321757 _runtime‚ 2074.66429544 _step‚97 0loss/FRt1/assignment_loss‚3.0374746546802815 1loss/FBt2/assignment_loss‚0.04227583631423641 .loss/FRt1/detection_loss‚0.608353029617112 1loss/SRqqt2/detection_loss‚0.8479384981448841 0loss/FBt1/detection_loss‚0.43935174617467637 % _timestamp‚1.7843319140791767e+09 epoch‚2X‘@ øƒ 1loss/SRqqt2/detection_loss‚0.8479384981448841 0loss/FBt1/detection_loss‚0.43935174617467637 % _timestamp‚1.7843319140791767e+09 epoch‚2 0loss/FRt2/assignment_loss‚2.1587146031856537 1loss/FBt1/assignment_loss‚0.11382521776747305 &loss/total_loss‚12.850717952633314 trainer/global_step‚1199 1loss/SRqqt1/assignment_loss‚1.142186361404745 2loss/SRqqt2/assignment_loss‚1.0117973919976138 /loss/FRt2/detection_loss‚1.4484863645549524 1loss/SRqqt1/detection_loss‚1.1717267866595096 /loss/FBt2/detection_loss‚0.8285874621321757 _runtime‚ 2074.66429544 _step‚97 0loss/FRt1/assignment_loss‚3.0374746546802815 1loss/FBt2/assignment_loss‚0.04227583631423641 .loss/FRt1/detection_loss‚0.608353029617112a¢ß‚žù‚(:“ ”ýêÒ⳯È gpu.0.gpu‚97.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚325.553' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.11059999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®½KF®ú‚(:£ ”ýêÒ÷›†Ê! disk.nvme0n1.in‚ 2765.50390625 disk.sda3.out‚ 63.171875 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.out‚5.671875) proc.memory.percent‚17.08998680114746 disk.dm-0.out‚ 9404.859375 disk.sda1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 2765.50390625 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 63.171875 disk.sda.in‚0$ memory_percent‚93.82057189941406 network.recv‚551869 disk.sda3.in‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.out‚ 9404.859375 network.sent‚ 2.114844e+06$ disk./.usageGB‚581.3376922607422 disk.dm-0.in‚ 2765.50390625 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 63.171875" proc.memory.rssMB‚ 2800.0234375 cpu‚25.457140183581554* disk./.usagePercent‚39.014060302431325 disk.nvme0n1.out‚ 9404.859375 disk.sda1.out‚0% proc.memory.availableMB‚ 1012.4375 proc.cpu.threads‚17ÑUœuû‚(:’ £ýêÒ²ëÓÇ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²¿ü‚(:´ £ýêÒÊ…ÙÊ disk.sda.in‚0 disk.sda.out‚ 64.9453125# disk.nvme0c0n1.in‚ 2765.53515625 proc.memory.rssMB‚ 2845.09375* disk./.usagePercent‚39.014608670569665$ disk.nvme0n1.out‚9420.2763671875 disk.sda3.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.365074157714844 cpu‚25.475542785724564 proc.cpu.threads‚17 network.sent‚ 2.177991e+06 disk.dm-1.out‚ 64.9453125& disk.nvme0c0n1.out‚9420.2763671875 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚94.09663677215576 disk.rbd0.in‚0 disk.rbd1.out‚5.671875 network.recv‚556695$ disk./.usageGB‚581.3458633422852 disk.sda2.in‚0! disk.dm-0.out‚9420.2763671875! disk.nvme0n1.in‚ 2765.53515625 disk.sda1.out‚0 disk.dm-0.in‚ 2765.53515625( proc.memory.availableMB‚ 967.20703125 disk.sda1.in‚0 disk.sda3.out‚ 64.9453125 disk.sda2.out‚0sì;2ý- _wandbruntime‚2102  _runtime‚2102hj) ‚þí{ 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12728598713874817 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3671875 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _runtime‚2102.638180494 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7230914831161499 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.04390435665845871 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 *jet/accuracy_3_of_3‚0.0513019785284996 (particle/accuracy‚0.8207968473434448 jet/accuracy_4_of_5‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6762133240699768 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.991976797580719 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9129493236541748 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967408180236816 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_1_of_4‚1 +particle/specificity‚0.8926879167556763 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5071492195129395 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9847134947776794 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.06054159626364708 'particle/f_score‚0.5721448063850403 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8907090425491333 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03512641414999962 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.10450716316699982 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16455724835395813 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9932998418807983 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9866716265678406 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.04390435665845871 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.08810998499393463 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4907894730567932 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6435185074806213 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6297429800033569 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.07296229153871536 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16346123814582825 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9951190948486328 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13846765458583832 *jet/accuracy_1_of_2‚0.6001562476158142 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1261594593524933 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09000000357627869 particle/accuracy_1_of_1‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 /particle/accuracy_3_of_3‚0.9990828633308411 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.234375 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.43748584389686584 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/sensitivity‚0.5579490661621094 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.12565693259239197 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4445485472679138 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4590025544166565 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8215005993 ÃÏq.843079 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9786577820777893 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4334026873111725 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5297406315803528 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9901484251022339 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9898673295974731 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.05750110000371933 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5009684562683105 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_5‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25579237937927246 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4647945761680603 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4763367176055908 ,jet/accuracy_4_of_4‚0.007130124606192112 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977155327796936 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17166836559772491 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.04208459332585335 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4911433160305023 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.613602340221405 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.021568866446614265 particle/accuracy_3_of_4‚1 6validation_average_jet_accuracy‚0.2436952292919159 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3611111044883728 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7311224937438965 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4423615038394928 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.48201555013656616 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5639524459838867 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 % _timestamp‚1.7843319420521123e+09 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 jet/accuracy_5_of_5‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5297406315803528 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.472610205411911 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4403471350669861 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2579510807991028 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.30650246143341064 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9922873377799988 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_2‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.05778554454445839 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4600888788700104 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4599670171737671 particle/accuracy_1_of_5‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8587899804115295 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4393405318260193 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3421187102794647 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9759637117385864 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10794173926115036 *jet/accuracy_2_of_4‚0.5453346967697144 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4662899374961853 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 +jet/accuracy_3_of_4‚0.06479257345199585 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 trainer/global_step‚1199 _step‚98 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4209192395210266 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4704904556274414 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9694271087646484 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15606243908405304 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7132275104522705 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.08082739263772964 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.515625 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.45625001192092896 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6435185074806213 +jet/accuracy_2_of_2‚0.09901069104671478 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.42973586916923523 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.13347534835338593 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.21296295523643494 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967408180236816 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.979477047920227 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.1840345859527588 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9974780082702637 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 *loss/val_total_loss‚13.053178524916788 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4819658398628235 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.49940231442451477 +jet/accuracy_1_of_1‚0.19789670407772064 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.46230679750442505 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8564814329147339 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.47027918696403503 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.05626838654279709 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9810245037078857 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9935869574546814 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.04567382112145424 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5021960139274597 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.49499520659446716 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.014589295722544193 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9908785223960876 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4858079254627228 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007542314473539591 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10794173926115036 *jet/accuracy_2_of_3‚0.3741702437400818 particle/accuracy_3_of_5‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.977704644203186 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9834466576576233 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0011886304710060358 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1767151653766632 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.04341060295701027 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7383787631988525 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0824408307671547 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4403471350669861 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.17873425781726837 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.08695439249277115 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15106289088726044 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5439712405204773 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.038127146661281586 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8322937488555908 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.413913369178772 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.08612094819545746 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.45837515592575073 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15055088698863983 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.17915232479572296 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9944124817848206 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.002192818559706211 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4236111044883728 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 jet/accuracy_3_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.07443170994520187 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4312010407447815 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9866756200790405 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.39884209632873535 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.7749064564704895 particle/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.21296295523643494 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.7887265086174011 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5027827620506287 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6291648149490356 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 epoch‚2 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.45512911677360535 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5271821618080139 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6866591572761536 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014477472752332687 6 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.136439248919487 /particle/accuracy_4_of_4‚0.9170408844947815îÕ¼ö÷=ÿñ{ =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.04208459332585335 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4911433160305023 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.613602340221405 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.021568866446614265 particle/accuracy_3_of_4‚1 6validation_average_jet_accuracy‚0.2436952292919159 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3611111044883728 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7311224937438965 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4423615038394928 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.48201555013656616 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5639524459838867 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 % _timestamp‚1.7843319420521123e+09 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 jet/accuracy_5_of_5‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5297406315803528 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.472610205411911 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4403471350669861 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2579510807991028 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.30650246143341064 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9922873377799988 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_2‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.05778554454445839 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4600888788700104 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4599670171737671 particle/accuracy_1_of_5‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8587899804115295 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4393405318260193 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3421187102794647 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9759637117385864 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10794173926115036 *jet/accuracy_2_of_4‚0.5453346967697144 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4662899374961853 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 +jet/accuracy_3_of_4‚0.06479257345199585 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 trainer/global_step‚1199 _step‚98 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4209192395210266 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4704904556274414 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9694271087646484 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15606243908405304 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7132275104522705 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.08082739263772964 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.515625 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.45625001192092896 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6435185074806213 +jet/accuracy_2_of_2‚0.09901069104671478 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.42973586916923523 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.13347534835338593 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.21296295523643494 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967408180236816 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.979477047920227 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.1840345859527588 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9974780082702637 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 *loss/val_total_loss‚13.053178524916788 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4819658398628235 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.49940231442451477 +jet/accuracy_1_of_1‚0.19789670407772064 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.46230679750442505 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8564814329147339 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.47027918696403503 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.05626838654279709 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9810245037078857 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9935869574546814 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.04567382112145424 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5021960139274597 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.49499520659446716 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.014589295722544193 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9908785223960876 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4858079254627228 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007542314473539591 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10794173926115036 *jet/accuracy_2_of_3‚0.3741702437400818 particle/accuracy_3_of_5‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.977704644203186 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9834466576576233 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0011886304710060358 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1767151653766632 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.04341060295701027 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7383787631988525 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0824408307671547 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4403471350669861 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.17873425781726837 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.08695439249277115 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15106289088726044 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5439712405204773 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.038127146661281586 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8322937488555908 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.413913369178772 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.08612094819545746 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.45837515592575073 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15055088698863983 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.17915232479572296 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9944124817848206 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.002192818559706211 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4236111044883728 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 jet/accuracy_3_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.07443170994520187 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4312010407447815 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9866756200790405 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.39884209632873535 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.7749064564704895 particle/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.21296295523643494 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.7887265086174011 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5027827620506287 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6291648149490356 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 epoch‚2 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.45512911677360535 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5271821618080139 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6866591572761536 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014477472752332687 6 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.136439248919487 /particle/accuracy_4_of_4‚0.9170408844947815 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12728598713874817 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3671875 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _runtime‚2102.638180494 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7230914831161499 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.04390435665845871 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 *jet/accuracy_3_of_3‚0.0513019785284996 (particle/accuracy‚0.8207968473434448 jet/accuracy_4_of_5‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6762133240699768 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.991976797580719 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9129493236541748 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967408180236816 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_1_of_4‚1 +particle/specificity‚0.8926879167556763 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5071492195129395 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9847134947776794 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.06054159626364708 'particle/f_score‚0.5721448063850403 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8907090425491333 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03512641414999962 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.10450716316699982 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16455724835395813 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9932998418807983 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9866716265678406 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.04390435665845871 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.08810998499393463 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4907894730567932 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6435185074806213 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6297429800033569 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.07296229153871536 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16346123814582825 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9951190948486328 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13846765458583832 *jet/accuracy_1_of_2‚0.6001562476158142 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1261594593524933 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09000000357627869 particle/accuracy_1_of_1‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 /particle/accuracy_3_of_3‚0.9990828633308411 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.234375 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.43748584389686584 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/sensitivity‚0.5579490661621094 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.12565693259239197 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4445485472679138 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4590025544166565 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8215005993843079 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9786577820777893 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4334026873111725 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5297406315803528 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9901484251022339 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9898673295974731 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.05750110000371933 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5009684562683105 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_5‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25579237937927246 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4647945761680603 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4763367176055908 ,jet/accuracy_4_of_4‚0.007130124606192112 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977155327796936 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17166836559772491 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355b ̺]Ÿ€ ‚(:” ²ýêÒ«Ú¶È gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚288.151' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.630199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›Ä†³ ‚(:¨ ²ýêÒЉÖÉ disk.sda1.out‚0 disk.sda3.in‚0$ memory_percent‚88.11378479003906 disk.sda.in‚0 disk.sda.out‚65.40625* proc.memory.percent‚17.368721961975098 disk.drbd2165.in‚0% disk.nvme0n1.out‚17298.1220703125$ disk./.usageGB‚588.0759162902832 disk.dm-1.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0% proc.memory.availableMB‚ 1947.4375# proc.memory.rssMB‚ 2845.69140625 disk.sda2.out‚0 disk.sda3.out‚65.40625 cpu‚25.47013224291572 proc.cpu.threads‚17) disk./.usagePercent‚39.46626817080036 disk.rbd1.out‚5.75' disk.nvme0c0n1.out‚17298.1220703125 disk.nvme0n1.in‚ 2766.5546875 disk.rbd0.out‚0 network.recv‚560044 disk.rbd1.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 2766.5546875" disk.dm-0.out‚17298.1220703125 network.sent‚ 2.182846e+06 disk.sda2.in‚0 disk.dm-1.out‚65.40625 disk.dm-0.in‚ 2766.5546875Ú"¦ó”‚ ‚(:‰ ÁýêÒƒÙÓÈ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚293.512' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.7024 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V»ý¸¼ƒ ‚(:± ÁýêÒ¨¹üÉ disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda.out‚ 66.015625' disk.nvme0c0n1.out‚17306.8466796875* proc.memory.percent‚17.373132705688477 network.recv‚564619 disk.rbd0.in‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 2767.38671875 cpu‚25.46940499990861! disk.nvme0n1.in‚ 2767.38671875% disk.nvme0n1.out‚17306.8466796875 disk.sda3.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 66.015625 disk.dm-0.in‚ 2767.38671875$ disk./.usageGB‚588.0759048461914 disk.sda1.out‚0" disk.dm-0.out‚17306.8466796875) proc.memory.availableMB‚ 1946.55859375" proc.memory.rssMB‚ 2846.4140625 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.out‚5.75$ memory_percent‚88.11914920806885 disk.sda3.out‚ 66.015625 proc.cpu.threads‚17 disk.drbd2165.out‚0 network.sent‚ 2.240401e+06) disk./.usagePercent‚39.46626740277776 disk.dm-1.in‚0e»ŽAŸ„ ‚(:” ÐýêÒ݃ìÈ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚291.919' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.383799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÒåÁ… ‚(:¶ ÐýêÒ°øòÉ disk.rbd0.out‚0! disk.nvme0n1.in‚ 2827.94140625 disk.dm-1.out‚ 66.37890625* proc.memory.percent‚17.373132705688477 disk.rbd1.in‚0 disk.sda.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0) disk./.usagePercent‚39.46627047486817 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 2827.94140625 cpu‚25.46869670255158% disk.nvme0n1.out‚17313.1982421875 disk.sda1.out‚0 disk.sda3.out‚ 66.37890625 disk.rbd1.out‚5.8125' disk.nvme0c0n1.out‚17313.1982421875" proc.memory.rssMB‚ 2846.4140625$ disk./.usageGB‚588.0759506225586# disk.nvme0c0n1.in‚ 2827.94140625 disk.sda.out‚ 66.37890625 disk.drbd2165.out‚0$ memory_percent‚88.12084197998047& proc.memory.availableMB‚ 1946.28125 disk.sda2.out‚0 network.sent‚ 2.245428e+06 network.recv‚567778 disk.sda3.in‚0 disk.rbd0.in‚0" disk.dm-0.out‚17313.1982421875 proc.cpu.threads‚17ÄöØ ž† ‚(:“ ßýêÒŠÇÖÇ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚292.743' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.54859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w˱7²‡ ‚(:ª ßýêÒ©’íÉ) disk./.usagePercent‚39.46623796191132 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 1946.99609375 network.recv‚571345% disk.nvme0n1.out‚17320.7373046875 disk.sda1.out‚0 disk.sda3.out‚ 66.703125 disk.rbd0.in‚0 disk.dm-1.out‚ 66.703125 disk.sda.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 2828.90625$ memory_percent‚8?7¶8.11647891998291" proc.memory.rssMB‚ 2845.6796875' disk.nvme0c0n1.out‚17320.7373046875 disk.sda2.in‚0 disk.drbd2165.out‚0 network.sent‚ 2.250352e+06$ disk./.usageGB‚588.0754661560059 disk.sda3.in‚0 disk.rbd1.out‚5.8125 disk.sda.out‚ 66.703125 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 2828.90625* proc.memory.percent‚17.368650436401367 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 2828.90625" disk.dm-0.out‚17320.7373046875 cpu‚25.46788297092436,š2cžˆ ‚(:“ îýêÒÃôÈÈ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚294.28' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.855999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aH³—À‰ ‚(:µ îýêÒܶƒÊ& disk.nvme0n1.out‚17367.04736328125 disk.sda2.out‚0 disk.dm-1.out‚ 67.18359375 disk.rbd1.in‚0 disk.sda1.out‚0 network.recv‚574594$ disk./.usageGB‚589.1800727844238( disk.nvme0c0n1.out‚17367.04736328125 disk.nvme0c0n1.in‚ 2828.90625 disk.dm-1.in‚0 disk.sda.out‚ 67.18359375$ memory_percent‚88.11707496643066( proc.memory.availableMB‚ 1946.8984375" proc.memory.rssMB‚ 2845.6796875 disk.sda3.in‚0) disk./.usagePercent‚39.54036903957115 disk.rbd1.out‚5.953125* proc.memory.percent‚17.368650436401367 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.sda3.out‚ 67.18359375 network.sent‚ 2.255878e+06 disk.dm-0.in‚ 2828.90625# disk.dm-0.out‚17367.04736328125 disk.nvme0n1.in‚ 2828.90625 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda.in‚0 cpu‚25.467188630975645ƒíÑ<”Š ‚(:‰ ýýêÒôž»Ç gpu.0.gpu‚45.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H¸:Ë ‚(:¸ ýýêÒêøÉ$ disk./.usageGB‚589.1412811279297 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 2829.73828125 disk.dm-0.in‚ 2829.73828125# disk.dm-0.out‚17374.49853515625 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2846.4140625 disk.dm-1.out‚ 67.421875* proc.memory.percent‚17.373132705688477& disk.nvme0n1.out‚17374.49853515625 disk.sda3.in‚0 disk.sda3.out‚ 67.421875 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.sda.out‚ 67.421875 disk.sda2.out‚0$ memory_percent‚88.12170028686523 cpu‚25.466622756931834 disk.rbd0.in‚0 disk.sda.in‚0' proc.memory.availableMB‚ 1946.140625 proc.cpu.threads‚17 network.sent‚ 2.260268e+06 network.recv‚577907* disk./.usagePercent‚39.537765698955454! disk.nvme0n1.in‚ 2829.73828125 disk.rbd1.out‚5.953125( disk.nvme0c0n1.out‚17374.49853515625L%‚2Œ - _wandbruntime‚2190  _runtime‚2190|ÆgúÆ À _runtime‚2190.135831783 _step‚99 & lr-AdamW/pg1‚0.0004802431847229895 & lr-AdamW/pg2‚0.0004802431847229895 trainer/global_step‚1249 % _timestamp‚1.7843320295479527e+09ÔÕg—ÊŽ Ä & lr-AdamW/pg1‚0.0004802431847229895 & lr-AdamW/pg2‚0.0004802431847229895 trainer/global_step‚1249 % _timestamp‚1.7843320295479527e+09 _runtime‚2190.135831783 _step‚99c8g¼È2 - _wandbruntime‚2190  _runtime‚2190°êKé' ãO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚360248425799168 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10827004 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3551535 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21484893 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10817031 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10657889 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10657883 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1176066612736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10817031 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚358916463349760 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10827004 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98710394284032 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360247054236160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚359072333012992 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1176066612736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360089147923300 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1172710001508 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _step‚100 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚359072333012992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1515893955584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚360248425799168 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1174721223168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97194097700864 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚358916437921792 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10657889 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97194465792000 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648gx;ýí'‘ çO YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10657889 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10657883 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1176066612736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10817031 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚358916463349760 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10827004 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98710394284032 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360247054236160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚359072333012992 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1176066612736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360089147923300 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1172710001508 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _step‚100 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚359072333012992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1515893955584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚360248425799168 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1174721223168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97194097700864 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚358916437921792 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10657889 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97194465792000 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚360248425799168 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10827004 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3551535 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21484893 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10817031dD/<2’ - _wandbruntime‚2191  _runtime‚2191PD ñ “ íM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚359350333743104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚2191.551275662 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚101 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10835496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359194319699968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚359350359916544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21501591 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1517101726720 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚1249 % _timestamp‚1.7843320309657013e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1173633467754 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21501591 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21491598 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10666095 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1174978068188 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3212251 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1175646258688 RA­ÿÊEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10666095 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97271952199680 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10825509 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6766296 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1517136256000 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚360527351571968 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820)Ùcˆ÷&” ñM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1173633467754 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21501591 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21491598 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10666095 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1174978068188 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3212251 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1175646258688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10666095 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97271952199680 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10825509 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6766296 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1517136256000 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚360527351571968 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚359350333743104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚2191.551275662 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚101 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10835496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359194319699968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚359350359916544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21501591 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1517101726720 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚1249 % _timestamp‚1.7843320309657013e+09 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7025565 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102644859594752 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101068786176000 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚372810552258560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11252756 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11242783 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1222382543360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚1299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3336858 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11068189 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372972369518592 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374194778235392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101068418084864 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1220291714938 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1221037153792 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3336853 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102644456967168 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374194778235392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 % _timestamp‚1.7843321128734903e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22320945 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11242783 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1222382543360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1218947114616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374193406672384 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3688712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _runtime‚2273.458987688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374194778235392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101068418084864 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1220291714938 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1221037153792 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3336853 ZFDeviceStatsMonitor.on_train_batch_stÝ#^pZ art/inactive_split_bytes.all.freed‚102644456967168 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374194778235392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 % _timestamp‚1.7843321128734903e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22320945 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11242783 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1222382543360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1218947114616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374193406672384 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3688712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _runtime‚2273.458987688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7025565 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102644859594752 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101068786176000 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚372810552258560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11252756 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11242783 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1222382543360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚1299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3336858 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568h*2¨ - _wandbruntime‚2274  _runtime‚2274•ƒJJö&© ðM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1221962189312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚101145904492544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1577234129920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7031091 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3339396 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1577268659200 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11261248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3691446 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374309649218032 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374473704008192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚373250370248704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22337643 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1219870580862 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11076389 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102723541242880 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3339401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374472332438016 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚374473704008192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11076395 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚374308279189630 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1299 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22327650 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22337643 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚373250370248704 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11261248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚105 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101146272583680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373088434036736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚373088408608768 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1221962189312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7030842 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3691690 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843321142856708e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373250396422144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1299 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22327650 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22337643 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚373250370248704 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11261248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚105 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101146272583680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373088434036736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚373088408608768 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1221962189312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7030842 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3691690 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843321142856708e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373250396422144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚101145904492544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1577234129920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7031091 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3339396 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1577268659200 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11261248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3691446 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374309649218032 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374473704008192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚373250370248704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22337643 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1219870580862 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11076389 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102723541242880 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995ikr³M2« - _wandbruntime‚2274  _runtime‚2274ûx é¬ ÿ 0loss/SRqqt1/assignment_loss‚1.02586154335378 1loss/FBt1/assignment_loss‚0.09392347332199551 0loss/SRqqt1/detection_loss‚1.167727355678965 1loss/SRqqt2/detection_loss‚0.7952571951760313 &loss/total_loss‚13.621941754420467 trainer/global_step‚1299 % _timestamp‚1.7843321143597112e+09 2loss/FBt2/assignment_loss‚0.008631759957893337 .loss/FRt2/detection_loss‚1.483299151380543 /loss/FBt2/detection_loss‚1.9366559511050583 0loss/FRt2/assignment_loss‚1.6148135973804298 /loss/FRt1/detection_loss‚0.5907640867862046 epoch‚3 _runtime‚2274.944799757 0loss/FRt1/assignment_loss‚2.9337570167807883 2loss/SRqqt2/assignment_loss‚0.8526532697097327 /loss/FBt1/detection_loss‚1.1185973537890461 _step‚106{p€ ­ ƒ 0loss/FRt2/assignment_loss‚1.6148135973804298 /loss/FRt1/detection_loss‚0.5907640867862046 epoch‚3 _runtime‚2274.944799757 0loss/FRt1/assignment_loss‚2.9337570167807883 2loss/SRqqt2/assignment_loss‚0.8526532697097327 /loss/FBt1/detection_loss‚1.1185973537890461 _step‚106 0loss/SRqqt1/assignment_loss‚1.02586154335378 1loss/FBt1/assignment_loss‚0.09392347332199551 0loss/SRqqt1/detection_loss‚1.167727355678965 1loss/SRqqt2/detection_loss‚0.7952571951760313 &loss/total_loss‚13.621941754420467 trainer/global_step‚1299 % _timestamp‚1.7843321143597112e+09 2loss/FBt2/assignment_loss‚0.008631759957893337 .loss/FRt2/detection_loss‚1.483299151380543 /loss/FBt2/detection_loss‚1.9366559511050583j©“bØ”® ‚(:‰ ×þêÒ¥‡äÇ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚278.726' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôàï ‚(:¸ ×þêÒúÉÁÉ& disk.nvme0n1.out‚18558.76123046875" proc.memory.rssMB‚ 2845.6640625 proc.cpu.threads‚17 network.recv‚598799 disk.sda1.out‚0 disk.dm-0.in‚ 2955.9296875# disk.dm-0.out‚18558.76123046875* proc.memory.percent‚17.368555068969727$ disk./.usageGB‚582.2480964660645 disk.sda1.in‚0 disk.sda.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 2955.9296875 disk.sda3.in‚0 disk.rbd0.in‚0 cpu‚25.46294705971964 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda.out‚ 73.67578125 disk.sda2.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 2955.9296875 disk.sda3.out‚ 73.67578125 disk.rbd1.in‚0 disk.rbd1.out‚6.15625' proc.memory.availableMB‚ 975.5859375 disk.dm-1.out‚ 73.67578125 network.sent‚ 2.348145e+06) disk./.usagePercent‚39.07515829253051 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚18558.76123046875$ memory_percent‚94.04549598693848n±%?”° ‚(:‰ æþêÒˆéóÇ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚280.902' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝJñ ‚(:¸ æþêÒ¬ ˆÊ disk.sda2.out‚0 disk.rbd1.out‚6.234375! disk.nvme0n1.in‚ 2956.89453125* proc.memory.percent‚17.368555068969727 cpu‚25.462419968067803 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2165.out‚0 network.recv‚602826* disk./.usagePercent‚39.075151636334624 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚18575.70556640625 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 75.609375# disk.nvme0c0n1.in‚ 2956.89453125' proc.memory.availableMB‚ 975.5859375 network.sent‚ 2.411293e+06& disk.nvme0n1.out‚18575.70556640625 disk.sda1.out‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 2845.6640625$ memory_percent‚94.04549598693848 disk.dm-1.out‚ 75.609375 disk.rbd1.in‚0 disk.dm-0.in‚ 2956.89453125# disk.dm-0.out‚18575.70556640625 disk.sda3.in‚0 disk.sda3.out‚ 75.609375$ disk./.usageGB‚582.2479972839355¤¿Õ%B² ‚(:‰ õþêÒª×ÑÈ gpu.0.gpu‚99.0 gpu.0.memory‚309ËÕER, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚280.902' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}œN±³ ‚(:¦ õþêÒ†ŸšÊ! disk.nvme0n1.in‚ 2956.89453125 disk.sda3.out‚76.15625$ disk./.usageGB‚582.2479476928711! disk.nvme0n1.out‚ 18580.109375 disk.sda2.in‚0$ memory_percent‚94.04709339141846 disk.drbd2165.in‚0 disk.dm-0.in‚ 2956.89453125* proc.memory.percent‚17.368555068969727 disk.dm-1.out‚76.15625 disk.rbd1.out‚6.234375 disk.sda.out‚76.15625 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 18580.109375( proc.memory.availableMB‚ 975.32421875 disk.rbd0.in‚0 disk.rbd1.in‚0 network.sent‚ 2.416269e+06 network.recv‚606349 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.drbd2165.out‚0 cpu‚25.461801582897433) disk./.usagePercent‚39.07514830823667 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 2956.89453125 disk.dm-0.out‚ 18580.109375" proc.memory.rssMB‚ 2845.6640625¤É¥‚Ÿ´ ‚(:” „ÿêÒÎüÃÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚285.446' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.089200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01‰Õ²µ ‚(:§ „ÿêÒì¥ãÉ! disk.nvme0c0n1.in‚ 3001.203125% disk.nvme0c0n1.out‚18593.94921875# disk.nvme0n1.out‚18593.94921875 cpu‚25.461204263810416 network.sent‚ 2.421201e+06 disk.rbd1.out‚6.359375$ memory_percent‚94.05183792114258 disk.rbd1.in‚0 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 77.1796875& proc.memory.availableMB‚ 974.546875 proc.cpu.threads‚17) disk./.usagePercent‚39.07495835064627 disk.sda2.out‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚609470 disk.dm-1.in‚0! disk./.usageGB‚582.2451171875 disk.sda2.in‚0 disk.dm-0.in‚ 3001.203125 disk.dm-0.out‚18593.94921875" proc.memory.rssMB‚ 2846.4140625* proc.memory.percent‚17.373132705688477 disk.dm-1.out‚ 77.1796875 disk.nvme0n1.in‚ 3001.203125 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda.out‚ 77.1796875 disk.drbd2165.in‚0„z”¶ ‚(:‰ “ÿêÒêÍàÇ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚275.808' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.1616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wZh· ‚(:· “ÿêÒå‹ôÉ disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.in‚0( proc.memory.availableMB‚ 975.78515625 disk.sda.out‚ 77.49609375 disk.dm-0.in‚ 3001.203125$ memory_percent‚94.04428005218506$ disk./.usageGB‚582.2451705932617 disk.sda3.in‚0 disk.sda.in‚0( disk.nvme0c0n1.out‚18596.77490234375 disk.rbd1.out‚6.359375 disk.nvme0n1.in‚ 3001.203125" proc.memory.rssMB‚ 2845.6484375) disk./.usagePercent‚39.07496193475175 disk.rbd0.out‚0 disk.dm-1.out‚ 77.49609375 disk.sda1.out‚0 cpu‚25.46060142765121 proc.cpu.threads‚17 disk.sda1.in‚0* proc.memory.percent‚17.368459701538086 network.sent‚ 2.425201e+06 network.recv‚612265 disk.rbd1.in‚0 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 3001.203125# disk.dm-0.out‚18596.77490234375& disk.nvme0n1.out‚18596.77490234375 disk.sda3.out‚ 77.49609375 disk.rbd0.in‚0 disk.dm-1.in‚0ª¯ÿiŸ¸ ‚(:” ¢ÿêÒà‡ñÈ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚305.928' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚61.185599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V"‡¢Å¹ ‚(:º ¢ÿêÒõžÝÉ( disk.nvme0c0n1.out‚18606.18212890625 disk.sda1.in‚0 disk.sda.out‚ 77.6953125" proc.memory.rssMB‚ 2845.6640625* proc.memory.percent‚17.368555068969727& disk.nvme0n1.out‚18606.18212890625 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚6.4375 network.sent‚ 2.430167e+06 disk.sda2.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 975.52734375* disk./.usagePercent‚39.074896140815454# disk.dm-0.out‚18606.18212890625 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0 network.recv‚615616# disk.nvme0c0n1.in‚ 3002.16796875 disk.dm-0.in‚ 3002.16796875 cpu‚25.459908568140314$ disk./.usageGB‚582.2441902160645! disk.nvme0n1.in‚ 3002.16796875 disk.rbd0.in‚0$ memory_percent‚94.04585361480713 disk.sda3.out‚ 77.6953125 disk.dm-1.out‚ 77.6953125 disk.drbd2165.in‚0 proc.cpu.threads‚17­ûŒ¾2º - _wandbruntime‚2356  _runtime‚2356ˆ€X;É» à ' lr-AdamW/pg1‚0.00047968794306184874 ' lr-AdamW/pg2‚0.00047968794306184874 trainer/global_step‚1349 % _timestamp‚1.7843321963764558e+09 _runtime‚2356.961803253 _step‚107wÃ\ͼ Ç % _timestamp‚1.7843321963764558e+09 _runtime‚2356.961803253 _step‚107 ' lr-AdamW/pg1‚0.00047968794306184874 ' lr-AdamW/pg2‚0.00047968794306184874 trainer/global_step‚1349kV:æ¤2½ - _wandbruntime‚2356  _runtime‚2356hx3{ì'¾ æO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚386872432197632 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11678508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3826045 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚388141130671616 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11478483 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386872406024192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1266528828046 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1268698473984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚386704615739392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _runtime‚2356.962747305 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚388139759108608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚108 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7290048 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1265184227724 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23147018 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388139759108608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388141130671616 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3464003 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387971169995406 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1636266981376 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3826275 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104942738468864 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387969799967116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1268698473984 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11478483 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23156997 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3464008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1267353084416 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388139759108608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388141130671616 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3464003 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387971169995406 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1636266981376 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3826275 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104942738468864 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387969799967116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1268698473984 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11478483 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23156997 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3464008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1267353084416 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386872406024192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1266528828046 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1268698473984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚386704615739392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _runtime‚2356.962747305 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚388139759108608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚108 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7290048 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1265184227724 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820lE±7Y2À - _wandbruntime‚2358  _runtime‚23580a IzÁ ñM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11486695 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387150406754304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚387150406754304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23173695 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚388249975239940 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7843321977972517e+09 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1269623516672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚221 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚388420056444416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23163702 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11687000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387150432927744 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3828707 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚109 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1267452294404 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11486689 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11677013 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7295253 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3828928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1637468218368 trainer/global_step‚1349 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23163702 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚388418684874240 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚386982522945536 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11486695 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3466546 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105020592967680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387150432927744 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚106657660662784 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚386982497517568 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106658061186048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105020224876544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175€@üû& õM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1267452294404 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11486689 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11677013 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7295253 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3828928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1637468218368 trainer/global_step‚1349 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23163702 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚388418684874240 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚386982522945536 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11486695 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3466546 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105020592967680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387150432927744 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚106657660662784 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚386982497517568 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106658061186048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105020224876544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11486695 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387150406754304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚387150406754304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23173695 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚388249975239940 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7843321977972517e+09 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1269623516672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚221 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚388420056444416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23163702 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11687000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387150432927744 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3828707 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚109 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648m1Æ®2à - _wandbruntime‚2358  _runtime‚2358_ÇŒôÄ  0loss/FRt2/assignment_loss‚1.5645890928664297 2loss/SRqqt1/assignment_loss‚1.1167722761518124 /loss/FBt1/detection_loss‚0.4822622824456827 /loss/FBt2/detection_loss‚1.2442182848049748 epoch‚3 trainer/global_step‚1349 1loss/FBt2/assignment_loss‚0.19296645647145536 .loss/FRt1/detection_loss‚0.565877147299095 /loss/FRt2/detection_loss‚1.4153729250144986 1loss/SRqqt2/detection_loss‚0.7899356927742456 &loss/total_loss‚12.312286386597027 0loss/FRt1/assignment_loss‚2.8408064052086917 2loss/SRqqt2/assignment_loss‚0.7420719009737442 1loss/FBt1/assignment_loss‚0.09566188207526798 1loss/SRqqt1/detection_loss‚1.2617520405111289 % _timestamp‚1.7843321978606322e+09 _runtime‚2358.446423581 _step‚110Êß«¯ Å … 0loss/FRt1/assignment_loss‚2.8408064052086917 2loss/SRqqt2/assignment_loss‚0.7420719009737442 1loss/FBt1/assignment_loss‚0.09566188207526798 1loss/SRqqt1/detection_loss‚1.2617520405111289 % _timestamp‚1.7843321978606322e+09 _runtime‚2358.446423581 _step‚110 0loss/FRt2/assignment_loss‚1.5645890928664297 2loss/SRqqt1/assignment_loss‚1.1167722761518124 /loss/FBt1/detection_loss‚0.4822622824456827 /loss/FBt2/detection_loss‚1.2442182848049748 epoch‚3 trainer/global_step‚1349 1loss/FBt2/assignment_loss‚0.19296645647145536 .loss/FRt1/detection_loss‚0.565877147299095 /loss/FRt2/detection_loss‚1.4153729250144986 1loss/SRqqt2/detection_loss‚0.7899356927742456 &loss/total_loss‚12.312286386597027nÍÝ/1žÆ ‚(:“ ±ÿêÒý¸óÈ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08¤èõ¢Ç ‚(:— ±ÿêÒ„ØÉ network.recv‚618768 disk.sda.in‚0 disk.nvme0c0n1.in‚3003! disk.dm-0.out‚18613.345703125 disk.drbd2165.in‚0 disk.dm-0.in‚3003 disk.nvme0n1.in‚3003 disk.sda1.in‚0 network.sent‚ 2.4351e+06 disk.sda3.out‚ 78.01953125 disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚94.04752254486084 disk.dm-1.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 975.25390625) disk./.usagePercent‚39.07489767686065 disk.sda2.in‚0 disk.dm-1.out‚ 78.01953125 cpu‚25.45942996098357# disk./.usageGB‚582.244213104248 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2845.6640625 proc.cpu.threads‚17 disk.rbd1.out‚6.4375 disk.sda.out‚ 78.01953125* proc.memory.percent‚17.368555068969727 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚18613.345703125$ disk.nvme0n1.out‚18613.345703125‹'…”È ‚(:‰ ÀÿêÒÐÓûÇ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚278.726' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\eÑg¾É ‚(:³ ÀÿêÒ¿ÃæÉ disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 3049.98828125 disk.sda1.out‚0 disk.sda.in‚0 network.sent‚ 2.498206e+06! disk.nvme0n1.in‚ 3049.98828125$ memory_percent‚94.05062198638916 cpu‚25.458850289914935) disk./.usagePercent‚39.07490586910175# disk.dm-0.out‚18618.43994140625 disk.sda3.out‚ 78.484375 network.recv‚622894 disk.sda2.out‚0 disk.dm-1.out‚ 78.484375 disk.rbd1.in‚0" proc.memory.rssMB‚ 2846.4140625 disk.sda.out‚ 78.484375* proc.memory.percent‚17.373132705688477 disk.dm-1.in‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚18618.43994140625 disk.sda1.in‚0( proc.memory.availableMB‚ 974.74609375# disk.nvme0c0n1.in‚ 3049.98828125( disk.nvme0c0n1.out‚18618.43994140625 proc.cpu.threads‚17 disk.rbd1.out‚6.5$ disk./.usageGB‚582.2443351745605 disk.sda2.in‚0 disk.sda3.in‚0RobÏŸÊ ‚(:” ÏÿêÒÈéúÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.062' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.812400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°¯Ù¸Ë ‚(:­ ÏÿêÒ¿®ÎÉ network.recv‚626181! disk.nvme0c0n1.in‚ 3050.953125 disk.nvme0n1.in‚ 3050.953125 disk.sda2.in‚0 disk.sda.in‚0 proc.memory.rssMB‚ 2845.46875) proc.memory.percent‚17.36736297607422 cpu‚25.458284103903928& disk.nvme0n1.out‚18626.97314453125 disk.rbd1.in‚0( proc.memory.availableMB‚ 975.34765625 disk.dm-0.in‚ 3050.953125# disk.dm-0.out‚18626.97314453125 disk.sda3.out‚ 78.84765625 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.dm-1.out‚ 78.84765625 disk.sda.out‚ 78.84765625) disk./.usagePercent‚39.07492071753874# disk./.usageGB‚582.244556427002 disk.rbd1.out‚6.5( disk.nvme0c0n1.out‚18626.97314453125 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 network.sent‚ 2.503117e+06" memory_percent‚94.046950340271 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0×£hô”Ì ‚(:‰ ÞÿêÒÅ×ßÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚264.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01ÍDÉÍ ‚(:¾ ÞÿêÒêŒìÉ network.sent‚ 2.508043e+06 network.recv‚629167) disk./.usagePercent‚39.07491303731271# disk.dm-0.out‚18639.38232421875 disk.sda.in‚0( disk.nvme0c0n1.out‚18639.38232421875! disk.nvme0n1.in‚ 3051.78515625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 79.02734375 disk.dm-0.in‚ 3051.78515625 disk.dm-1.out‚ 79.02734375 disk.rbd1.out‚6.578125 disk.sda1.in‚0 disk.sda3.in‚0$ memory_percent‚94.04685497283936# proc.memory.rssMB‚ 2845.66796875 proc.cpu.threads‚17( proc.memory.availableMB‚ 975.36328125 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 3051.78515625 disk.drbd2165.out‚0 cpu‚25.457715777709993# disk./.usageGB‚582.244441986084& disk.nvme0n1.out‚18639.38232421875 disk.sda1.out‚0 disk.sda3.out‚ 79.02734375 disk.sda2.in‚0* proc.memory.percent‚17.368578910827637«ÛãE”Î ‚(:‰ íÿêÒ¿ÉÈÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BþøËÅÏ ‚(:º íÿêÒÂÊÎÉ disk.sda1.in‚0$ memory_percent‚94.04699802398682 disk.rbd0.out‚0 disk.sda.in‚0 cpu‚25.457164506485576 disk.sda2.in‚0 disk.dm-1.out‚ 79.484375 disk.rbd1.in‚0 disk.rbd1.out‚6.578125 disk.sda.out‚ 79.484375 disk.sda3.out‚ 79.484375 disk.rbd0.in‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2845.66796875 network.recv‚632813$ disk./.usageGB‚582.2434310913086! disk.nvme0n1.in‚ 3051.78515625 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 3051.78515625* proc.memory.percent‚17.368578910827637 network.sent‚ 2.513016e+06 disk.sda2.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚18643.76416015625 disk.dm-0.in‚ 3051.78515625* disk./.usagePercent‚39.074845195316136& disk.nvme0n1.out‚18643.76416015625 disk.sda3.in‚0# disk.dm-0.out‚18643.76416015625( proc.memory.availableMB‚ 975.33984375¢©ß´2Ð - _wandbruntime‚2440  _runtime‚2440tª”nÇÑ Á & lr-AdamW/pg2‚0.0004793873401527131 trainer/global_step‚1399 % _timestamp‚1.7843322799639535e+09 _runtime‚2440.549269969 _step‚111 & lr-AdamW/pg1‚0.0004793873401527131:‡·ËÒ Å & lr-AdamW/pg1‚0.0004793873401527131 & lr-AdamW/pg2‚0.0004793873401527131 trainer/global_step‚1399 % _timestamp‚1.7843322799639535e+09 _runtime‚2440.549269969 _step‚111oÖÞFþ2Ó - _wandbruntime‚2440  _runtime‚2440*&#'Ô çO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11888789 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7554414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23983070 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1315014404608 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12104260 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3591158 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108817426944000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110513397756928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12094287 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1313669015040 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23993049 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12104260 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚402087483107840 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843322799657521e+09 _runtime‚2440.552163573 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11888783 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3962995 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108817058852864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7554148 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3591153 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚402087483107840 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1313669015040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1696373440512 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚401910125989024 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚402086111544832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚400598730076160 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400772468703232 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚402086111544832 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1311421340832 _step‚112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400772442529792 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚400598704648192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11888783 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚401911496017314 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1315014404608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚1399 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108817058852864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7554148 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3591153 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚402087483107840 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1313669015040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1696373440512 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚401910125989024 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚402086111544832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚400598730076160 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400772468703232 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚402086111544832 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1311421340832 _step‚112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400772442529792 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚400598704648192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11888783 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚401911496017314 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1315014404608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚1399 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12104260 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚402087483107840 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843322799657521e+09 _runtime‚2440.552163573 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11888783 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3962995 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0pšL32Ö - _wandbruntime‚2441  _runtime‚2441,ïFö&× ðM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3966005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚400876611854336 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1313689407512 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11896989 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3965757 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚400876586426368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1315939447296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12112752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚2441.971417534 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1314594050560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1312344807078 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24009747 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110592484596736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12102765 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11896989 $ _timestamp‚1.784332281385854e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚402365037310464 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1314594050560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23999754 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11896995 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12102765 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _step‚113 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23999754 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24009747 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11896995 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚401050469433344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1697571245056 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚402190301261848 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚402366408880640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚402366408880640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7559453 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0‰ù4ú&Ø ôM J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7559453 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3966005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚400876611854336 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1313689407512 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11896989 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3965757 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚400876586426368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1315939447296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12112752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚2441.971417534 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1314594050560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1312344807078 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24009747 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110592484596736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12102765 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11896989 $ _timestamp‚1.784332281385854e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚402365037310464 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1314594050560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23999754 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11896995 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12102765 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _step‚113 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23999754 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24009747 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11896995 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚401050469433344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1697571245056 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚402190301261848 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚402366408880640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚402366408880640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚414492793556992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _step‚116 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚414672479035392 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4101393 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12520039 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1361330335232 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114448006610944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚416032463981056 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚2523.512432859 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12520039 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚1449 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24829101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3718308 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112691379236864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚415850452010932 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24819122 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚416032463981056 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1359984945664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926É%cÓñ'í ëO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚415850452010932 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24819122 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚416032463981056 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1359984945664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1359984945664 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414492818984960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚416033835544064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 % _timestamp‚1.7843323629272213e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1756259282944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12299083 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3718303 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚415851822039222 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12299089 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7819701 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1359003054262 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚414492793556992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _step‚116 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚414672479035392 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4101393 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12520039 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1361330335232 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114448006610944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚416032463981056 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚2523.512432859 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12520039 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚1449 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24829101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3718308 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112691379236864te˶ 2î - _wandbruntime‚2524  _runtime‚2524%Ðùžðï ñM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1359926520620 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 % _timestamp‚1.7843323643398745e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114526679132672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚416129257255354 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24845799 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚416312761316864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114526274415104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚416311389746688 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1360909981184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4103849 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚416130627283756 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12307289 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7824955 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1362255377920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12307295 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚414770675335168 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚414950479765504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3720851 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3720846 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚1449 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24845799 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12307295 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112768865644544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1757408770560 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚117 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1360909981184 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7824695 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1757445396992 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1358581920186 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440ìkw‰û&ð õM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1757408770560 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚117 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1360909981184 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7824695 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1757445396992 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1358581920186 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1359926520620 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 % _timestamp‚1.7843323643398745e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114526679132672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚416129257255354 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24845799 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚416312761316864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114526274415104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚416311389746688 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1360909981184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4103849 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚416130627283756 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12307289 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7824955 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1362255377920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12307295 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚414770675335168 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚414950479765504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3720851 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3720846 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚1449 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24845799 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12307295 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112768865644544uý»„.2ñ - _wandbruntime‚2524  _runtime‚2524 ØJÀò þ 1loss/FBt1/assignment_loss‚0.03323117770428921 /loss/FRt1/detection_loss‚0.5823241027345086 1loss/SRqqt1/detection_loss‚1.1786733470733994 trainer/global_step‚1449 epoch‚3 -loss/FRt1/assignment_loss‚2.6201280529729 2loss/SRqqt1/assignment_loss‚0.9981903018553026 /loss/FRt2/detection_loss‚1.6808308519400152 1loss/SRqqt2/detection_loss‚0.7727188916454456 /loss/FBt2/detection_loss‚1.3740202464784186 &loss/total_loss‚11.748186154955217 % _timestamp‚1.7843323644113755e+09 1loss/FBt2/assignment_loss‚0.02854434051440761 /loss/FBt1/detection_loss‚0.5508834646989395 _step‚118 /loss/FRt2/assignment_loss‚1.228791040389328 2loss/SRqqt2/assignment_loss‚0.6998503369482633 _runtime‚2524.996466837˜{…™ó ‚ trainer/global_step‚1449 epoch‚3 -loss/FRt1/assignment_loss‚2.6201280529729 2loss/SRqqt1/assignment_loss‚0.9981903018553026 /loss/FRt2/detection_loss‚1.6808308519400152 1loss/SRqqt2/detection_loss‚0.7727188916454456 /loss/FBt2/detection_loss‚1.3740202464784186 &loss/total_loss‚11.748186154955217 % _timestamp‚1.7843323644113755e+09 1loss/FBt2/assignment_loss‚0.02854434051440761 /loss/FBt1/detection_loss‚0.5508834646989395 _step‚118 /loss/FRt2/assignment_loss‚1.228791040389328 2loss/SRqqt2/assignment_loss‚0.6998503369482633 _runtime‚2524.996466837 1loss/FBt1/assignment_loss‚0.03323117770428921 /loss/FRt1/detection_loss‚0.5823241027345086 1loss/SRqqt1/detection_loss‚1.1786733470733994v6L{žô ‚(:“ Ö€ëÒÍäêÇ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚264.23' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.846000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0["’€Äõ ‚(:¹ Ö€ëÒÛåÉ) disk./.usagePercent‚39.07487617222778 disk.sda2.out‚0 disk.sda.out‚ 81.82421875$ memory_percent‚94.04761791229248 disk.dm-1.in‚0$ disk.nvme0n1.out‚18696.650390625 disk.dm-1.out‚ 81.82421875 disk.dm-0.in‚ 3203.16015625 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 81.82421875 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 3203.16015625& disk.nvme0c0n1.out‚18696.650390625! disk.nvme0n1.in‚ 3203.16015625 disk.rbd1.out‚6.921875 disk.drbd2165.out‚0 cpu‚25.453656206967356 disk.sda2.in‚0! disk.dm-0.out‚18696.650390625( proc.memory.availableMB‚ 975.23828125* proc.memory.percent‚17.368578910827637 network.recv‚657508$ disk./.usageGB‚582.2438926696777 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 2.605915e+06 disk.sda1.in‚0 disk.sda.in‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2845.66796875 proc.cpu.threads‚17YÅ ²Ÿö ‚(:” å€ëÒ¦¯¥Ç gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚284.671' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.934200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oØZ$Ë÷ ‚(:À å€ëÒò€‘Ê network.recv‚661844 disk.rbd0.out‚0 disk.dm-0.in‚ 3203.16015625 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚18699.90771484375# proc.memory.rssMB‚ 2845.66796875* proc.memory.percent‚17.368578910827637 proc.cpu.threads‚17 disk.rbd0.in‚0& disk.nvme0n1.out‚18699.90771484375! disk.nvme0n1.in‚ 3203.16015625$ disk./.usageGB‚582.2440338134766 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚6.921875# disk.dm-0.out‚18699.90771484375 cpu‚25.453130802116828 network.sent‚ 2.668959e+06* disk./.usagePercent‚39.074885644506544 disk.sda2.out‚0 disk.sda.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 3203.16015625 disk.sda3.in‚0$ memory_percent‚94.04761791229248 disk.dm-1.out‚ 82.08203125( proc.memory.availableMB‚ 975.23828125 disk.sda3.out‚ 82.08203125 disk.sda.out‚ 82.08203125 disk.drbd2165.in‚0pÑX+Ÿø ‚(:” ô€ëÒ¥©ÂÈ gpu.0.gpu‚95.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚293.457' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.691399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]I.ÙÁù ‚(:¶ ô€ëÒúõôÈ" disk.dm-0.out‚18709.5205078125 disk.sda.in‚0' disk.nvme0c0n1.out‚18709.5205078125* proc.memory.percent‚17.373156547546387 cpu‚25.45275184613947 disk.drbd2165.out‚0 disk.dm-1.out‚ 82.26953125 disk.sda3.in‚0 network.recv‚665022 disk.sda1.in‚0& proc.memory.availableMB‚ 974.484375 disk.rbd1.out‚6.984375* disk./.usagePercent‚39.074885132491474 disk.sda2.out‚0 disk.dm-0.in‚ 3266.2421875 disk.nvme0n1.in‚ 3266.2421875 disk.sda3.out‚ 82.26953125# proc.memory.rssMB‚ 2846.41796875 disk.drbd2165.in‚0% disk.nvme0n1.out‚18709.5205078125 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 2.673795e+06# disk./.usageGB‚582.244026184082 disk.dm-1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0$ memory_percent‚94.05221939086914 disk.sda.out‚ 82.26953125 disk.sda2.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 3266.2421875îçŸú ‚(:” ƒëÒœŒ¤Ç gpu.0.gpu‚84.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚284.382' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.876400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â^»¹Åû ‚(:º ƒëÒáìÿÈ proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda3.in‚0 disk.dm-0.in‚ 3267.20703125( proc.memory.availableMB‚ 976.47265625 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0" disk.dm-0.out‚18716.7939453125$ memory_percent‚94.04008388519287$ disk./.usageGB‚582.2442626953125 disk.rbd1.out‚6.984375 network.sent‚ 2.67879e+06 disk.sda.in‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚18716.7939453125 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚668677 disk.dm-1.out‚ 82.66015625 disk.rbd1.in‚0# proc.memory.rssMB‚ 2844.41796875 disk.sda3.out‚ 82.66015625* proc.memory.percent‚17.360949516296387# disk.nvme0c0n1.in‚ 3267.20703125( disk./.usagePercent‚39.0749010049586! disk.nvme0n1.in‚ 3267.20703125% disk.nvme0n1.out‚18716.7939453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 82.66015625 cpu‚25.452158337608484 ã¡™”ü ‚(:‰ ’ëÒá´µÇ gpu.0.gpu‚93.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚288.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.7936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ž~Áý ‚(:¶ ’ëÒòÇÄÉ$ disk.nvme0n1.out‚18722.904296875 disk.sda1.out‚0& disk.nvme0c0n1.out‚18722.904296875 disk.sda3.out‚ 82.91015625 disk.rbd0.out‚0 disk.sda.out‚ 82.91015625 disk.dm-0.in‚ 3267.20703125! disk.nvme0n1.in‚ 3267.20703125 disk.dm-1.in‚0 disk.dm-1.out‚ 82.91015625 disk.sda.in‚0( proc.memory.availableMB‚ 975.45703125 network.recv‚672028 disk.sda2.in‚0# disk./.usageGB‚582.244270324707 disk.sda1.in‚0 disk.rbd1.out‚7.0625 disk.drbd2165.out‚0) disk./.usagePercent‚39.07490151697367! disk.dm-0.out‚18722.904296875 disk.sda2.out‚0# proc.memory.rssMB‚ 2845.66796875 cpu‚25.451753407780988 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd1.in‚0$ memory_percent‚94.04628276824951 network.sent‚ 2.683833e+06# disk.nvme0c0n1.in‚ 3267.20703125 disk.rbd0.in‚0 disk.drbd2165.in‚0* proc.memory.percent‚17.368578910827637Ž{eN2þ - _wandbruntime‚2607  _runtime‚2607<Â8ýÅÿ ¿ _runtime‚2607.569738945 _step‚119 % lr-AdamW/pg1‚0.000478740266711887 % lr-AdamW/pg2‚0.000478740266711887 trainer/global_step‚1499 % _timestamp‚1.7843324469841433e+093»áÒÉ€ à % lr-AdamW/pg2‚0.000478740266711887 trainer/global_step‚1499 % _timestamp‚1.7843324469841433e+09 _runtime‚2607.569738945 _step‚119 % lr-AdamW/pg1‚0.000478740266711887wþûçÎ2 - _wandbruntime‚2607  _runtime‚2607<ç_È]‚ çO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1407646265856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3845453 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2607.572367405 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4237731 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚428572515540992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚428572541714432 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1405240167370 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429980187980288 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3845458 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚118381491264512 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429978816417280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8083184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12709383 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1815826180096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116565699620864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚428386882465792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚428572541714432 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116566067712000 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12945791 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚428572515540992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12709389 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12955764 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12945791 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚429978816417280 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1815791643648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12709383 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_by¯ÞÔ tes.small_pool.current‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25655174 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1406300876288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚429790778032840 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚428386907893760 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12709389 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚118381893892096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚428386882465792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚428572541714432 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116566067712000 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12945791 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚428572515540992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12709389 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12955764 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12945791 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚429978816417280 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1815791643648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12709383 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25655174 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1406300876288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚429790778032840 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚428386907893760 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12709389 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚118381893892096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429978816417280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0xÒlý2„ - _wandbruntime‚2608  _runtime‚2608*(s¿÷&… ñM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚428850542444544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12717595 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1408571308544 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25671858 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4240464 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12717595 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12717589 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚118460185822208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚430259113753088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12717589 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8088460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1499 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8088706 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚428664764243968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1406163633728 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12954269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3848001 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12964256 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12954269 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1404819033294 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚428850542444544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116643186028544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚428850516271104 % _timestamp‚1.7843324484089434e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25671858 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚428850516271104 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1407225911808 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25681851 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116643554119680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1817032225792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1816999793664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6ÿ-œLû&† õM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚428850542444544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116643186028544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚428850516271104 % _timestamp‚1.7843324484089434e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25671858 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚428850516271104 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1407225911808 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25681851 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116643554119680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1817032225792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1816999793664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚428850542444544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12717595 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1408571308544 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25671858 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4240464 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12717595 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12717589 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚118460185822208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚430259113753088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12717589 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8088460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1499 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8088706 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚428664764243968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1406163633728 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12954269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3848001 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12964256 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12954269 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1404819033294 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640yjAŽÇ1‡ - _wandbruntime‚2609  _runtime‚260ó_°z9^Y<Òˆ þ 1loss/SRqqt1/detection_loss‚1.3705141081317376 /loss/FRt2/detection_loss‚1.7205545491494205 /loss/FBt1/detection_loss‚0.5562823201249634 /loss/FBt2/detection_loss‚1.5751891718448505 trainer/global_step‚1499 .loss/FRt1/assignment_loss‚2.57480929904701 0loss/FRt2/assignment_loss‚1.0168451328464991 1loss/FBt2/assignment_loss‚0.06382168082515767 /loss/FRt1/detection_loss‚0.6297330991328892 1loss/SRqqt2/detection_loss‚0.7801102983954334 _step‚122 2loss/SRqqt2/assignment_loss‚0.6981202462197645 epoch‚3 % _timestamp‚1.7843324484793584e+09 _runtime‚2609.064486405 %loss/total_loss‚12.02286674811705 2loss/SRqqt1/assignment_loss‚0.9714368244252469 0loss/FBt1/assignment_loss‚0.0654500179740773»ˆþ‰ ‚ /loss/FBt2/detection_loss‚1.5751891718448505 trainer/global_step‚1499 .loss/FRt1/assignment_loss‚2.57480929904701 0loss/FRt2/assignment_loss‚1.0168451328464991 1loss/FBt2/assignment_loss‚0.06382168082515767 /loss/FRt1/detection_loss‚0.6297330991328892 1loss/SRqqt2/detection_loss‚0.7801102983954334 _step‚122 2loss/SRqqt2/assignment_loss‚0.6981202462197645 epoch‚3 % _timestamp‚1.7843324484793584e+09 _runtime‚2609.064486405 %loss/total_loss‚12.02286674811705 2loss/SRqqt1/assignment_loss‚0.9714368244252469 0loss/FBt1/assignment_loss‚0.0654500179740773 1loss/SRqqt1/detection_loss‚1.3705141081317376 /loss/FRt2/detection_loss‚1.7205545491494205 /loss/FBt1/detection_loss‚0.5562823201249634zl?1”Š ‚(:‰ ¡ëÒÀÑêÇ gpu.0.gpu‚89.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚270.126' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0252 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ[ø¾‹ ‚(:³ ¡ëÒƒñ£Ê disk.sda2.in‚0 disk.nvme0n1.in‚ 3268.0390625& disk.nvme0n1.out‚18730.65380859375# memory_percent‚94.0469741821289 disk.sda.in‚0% proc.memory.availableMB‚ 975.34375 network.sent‚ 2.688822e+06) disk./.usagePercent‚39.07492967780243 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 network.recv‚675469" disk.nvme0c0n1.in‚ 3268.0390625 disk.sda3.out‚ 83.3046875 disk.rbd1.in‚0 disk.rbd1.out‚7.0625 disk.sda.out‚ 83.3046875# proc.memory.rssMB‚ 2845.66796875 disk.sda1.in‚0* proc.memory.percent‚17.368578910827637$ disk./.usageGB‚582.2446899414062 disk.sda3.in‚0 disk.dm-1.out‚ 83.3046875 cpu‚25.451254321112906 disk.drbd2165.out‚0 disk.dm-0.in‚ 3268.0390625 disk.rbd0.in‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚18730.65380859375# disk.dm-0.out‚18730.65380859375 disk.sda1.out‚0s•p꾌 ‚(:³ °ëÒˆæ¹É disk.dm-1.in‚0 disk.dm-1.out‚ 83.5546875 disk.nvme0n1.in‚ 3314.8515625 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚94.05350685119629$ disk./.usageGB‚582.2445259094238* disk./.usagePercent‚39.074918669478464 disk.sda2.in‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚18742.5380859375 proc.cpu.threads‚17 network.recv‚679796 network.sent‚ 2.751858e+06 disk.sda3.out‚ 83.5546875 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 3314.8515625# proc.memory.rssMB‚ 2846.41796875 cpu‚25.450886299761763 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 83.5546875" disk.dm-0.out‚18742.5380859375* proc.memory.percent‚17.373156547546387 disk.rbd0.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 3314.8515625% disk.nvme0n1.out‚18742.5380859375' proc.memory.availableMB‚ 974.2734375 disk.sda2.out‚0 disk.rbd1.out‚7.125àôõ Ÿ ‚(:” °ëÒºýÌÓ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.015' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.803000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`T”Ž ‚(:‰ ¿ëÒ§¼©Ç gpu.0.gpu‚86.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚248.861' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.7722 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>#¬» ‚(:° ¿ëÒèþÉ) disk./.usagePercent‚39.07492148556134! disk.dm-0.out‚18745.779296875* proc.memory.percent‚17.373156547546387$ disk./.usageGB‚582.2445678710938 disk.dm-1.out‚ 83.87890625 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 3314.8515625 disk.nvme0n1.in‚ 3314.8515625 disk.sda2.in‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚18745.779296875 network.recv‚683101 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 83.87890625 disk.dm-0.in‚ 3314.8515625 disk.sda3.out‚ 83.87890625 disk.rbd0.out‚0% proc.memory.availableMB‚ 973.96875 disk.drbd2165.in‚0$ memory_percent‚94.05536651611328 network.sent‚ 2.756784e+06$ disk.nvme0n1.out‚18745.779296875 disk.sda1.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.out‚7.125# proc.memory.rssMB‚ 2846.41796875 cpu‚25.450411537821715 proc.cpu.threads‚17µ‰9” ‚(:‰ ÎëÒáÃêÇ gpu.0.gpu‚82.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚253.749' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7498 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”&¼€Ä‘ ‚(:¹ ÎëÒÖ±¾É network.recv‚686790 disk.dm-1.in‚0& disk.nvme0c0n1.out‚18755.005859375 disk.dm-0.in‚ 3315.68359375! disk.nvme0n1.in‚ 3315.68359375 disk.sda1.in‚0# proc.memory.rssMB‚ 2846.41796875 disk.sda.out‚ 84.12890625! disk.dm-0.out‚18755.005859375$ memory_percent‚94.05531883239746 disk.rbd1.out‚7.203125 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.rbd0.out‚0* disk./.usagePercent‚39.074923277614076# disk.nvme0c0n1.in‚ 3315.68359375 disk.sda1.out‚0* proc.memory.percent‚17.373156547546387$ disk./.usageGB‚582.2445945739746 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 84.12890625 disk.rbd1.in‚0$ disk.nvme0n1.out‚18755.005859375 disk.sda3.out‚ 84.12890625 proc.cpu.threads‚17 network.sent‚ 2.761767e+06 disk.sda2.out‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 973.9765625 cpu‚25.449942964460597 disk.drbd2165.in‚0Òãj”’ ‚(:‰ ÝëÒÞ¡ÔÇ gpu.0.gpu‚86.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ³iÈ“ ‚(:½ ÝëÒݦÓÉ$ disk./.usageGB‚582.2447395324707 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚7.203125 disk.drbd2165.in‚0 disk.sda.in‚0 disk.rbd1.in‚0$ memory_percent‚94.05536651611328* disk./.usagePercent‚39.074933005900384# disk.nvme0c0n1.in‚ 3315.68359375& disk.nvme0n1.out‚18758.18310546875 disk.sda1.out‚0# proc.memory.rssMB‚ 2846.41796875# disk.dm-0.out‚18758.18310546875! disk.nvme0n1.in‚ 3315.68359375 disk.sda2.in‚0 disk.dm-1.in‚0 network.sent‚ 2.766688e+06 disk.dm-0.in‚ 3315.68359375 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 84.76171875 cpu‚25.449568947670635 network.recv‚690466 disk.drbd2165.out‚0 disk.sda3.in‚0% proc.memory.availableMB‚ 973.96875 proc.cpu.threads‚17 disk.sda.out‚ 84.76171875( disk.nvme0c0n1.out‚18758.18310546875 disk.sda3.out‚ 84.76171875* proc.memory.percent‚17.373156547546387†m”” ‚(:‰ ìëÒ·¬¦Ç gpu.0.gpu‚81.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚255.341' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.0682 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02ùàx¾• ‚(:³ ìëÒãŠË) disk./.usagePercent‚39.07491278130518 disk.dm-1.out‚ 85.0390625 disk.rbd1.out‚7.265625! disk.nvme0c0n1.in‚ 3363.578125 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 3363.578125# disk.dm-0.out‚18768.27099609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 85.0390625 disk.drbd2165.out‚0$ memory_percent‚94.05536651611328% proc.memory.availableMB‚ 973.96875 network.sent‚ 2.773634e+06 network.recv‚693694# proc.memory.rssMB‚ 2846.41796875 cpu‚25.448989136964897$ disk./.usageGB‚582.2444381713867 disk.sda3.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚18768.27099609375( disk.nvme0c0n1.out‚18768.27099609375 disk.nvme0n1.in‚ 3363.578125 disk.sda3.out‚ 85.0390625 disk.sda2.in‚0 disk.rbd1.in‚0* proc.memory.percent‚17.3731565475463879ï˜J2– - _wandbruntime‚2691  _runtime‚2691ÒÕÕÃÇ— Á & lr-AdamW/pg1‚0.0004783938377405145 & lr-AdamW/pg2‚0.0004783938377405145 trainer/global_step‚1549 % _timestamp‚1.7843325304733145e+09 _runtime‚2691.058817864 _step‚123C(ÖÐ˘ Å & lr-AdamW/pg1‚0.0004783938377405145 & lr-AdamW/pg2‚0.0004783938377405145 trainer/global_step‚1549 % _timestamp‚1.7843325304733145e+09 _runtime‚2691.058817864 _step‚123{ÜCñ2™ - _wandbruntime‚2691  _runtime‚2691S}Õí'š çO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122316253912064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3972603 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122316654442496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚443731104054748 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26501205 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13119683 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443926540416512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚443732474083038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443925168853504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1876233907200 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚1549 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚443926540416512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442472552046592 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13371543 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4375801 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13381516 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442472552046592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1452616806912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8348650 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3972608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1451477280478 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26491226 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120440020004864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚442280996802560 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚442280971374592 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1450132680156 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1452616806912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚442472578220032 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1453962196480 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚442472578220032 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443925168853504 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4376042 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1876266346496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚124 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13119683 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120440388096000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚2691.059878025{°ˆñ'› ëO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120440388096000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚2691.059878025 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122316253912064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3972603 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122316654442496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚443731104054748 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26501205 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13119683 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443926540416512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚443732474083038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443925168853504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1876233907200 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚1549 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚443926540416512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442472552046592 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13371543 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4375801 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13381516 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442472552046592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1452616806912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8348650 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3972608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1451477280478 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26491226 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120440020004864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚442280996802560 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚442280971374592 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1450132680156 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1452616806912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚442472578220032 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1453962196480 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚442472578220032 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443925168853504 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4376042 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1876266346496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚124 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13119683|0Æ2œ - _wandbruntime‚2692  _runtime‚2692ôŽæ¶H ñM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13127889 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26507910 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13127895 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3975151 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120517874503680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13390008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13127889 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8353679 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1877470237696 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3975146 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚442558878580736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1454887239168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _step‚125 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26507910 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1549 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚444205466189312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1453541842432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13127895 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26517903 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26517903 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995bÍß…û&ž õM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26517903 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13127889 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26507910 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13127895 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3975151 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120517874503680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13390008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13127889 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8353679 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1877470237696 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3975146 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚442558878580736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1454887239168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _step‚125 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26507910 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1549 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚444205466189312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1453541842432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13127895 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26517903 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175}D±‡R2Ÿ - _wandbruntime‚2692  _runtime‚2692ðÈ›§ÿ  ù 2loss/SRqqt1/assignment_loss‚0.8727616525299301 -loss/FRt1/detection_loss‚0.63008733588358 /loss/FBt2/detection_loss‚1.3094623336801305 _runtime‚ 2692.55280677 0loss/FRt1/assignment_loss‚2.4422356865453274 1loss/SRqqt2/assignment_loss‚0.751899297711584 /loss/FBt2/assignment_loss‚0.214468859839112 .loss/FRt2/detection_loss‚1.718841116516346 %loss/total_loss‚11.76662479528002 _step‚126 /loss/FRt2/assignment_loss‚1.287490599246389 1loss/SRqqt1/detection_loss‚1.1621776901442429 % _timestamp‚1.7843325319676712e+09 1loss/FBt1/assignment_loss‚0.09425421339628827 1loss/SRqqt2/detection_loss‚0.7325334584497211 /loss/FBt1/detection_loss‚0.5504125513373695 epoch‚3 trainer/global_step‚1549î ¾¡ ý 0loss/FRt1/assignment_loss‚2.4422356865453274 1loss/SRqqt2/assignment_loss‚0.751899297711584 /loss/FBt2/assignment_loss‚0.214468859839112 .loss/FRt2/detection_loss‚1.718841116516346 %loss/total_loss‚11.76662479528002 _step‚126 /loss/FRt2/assignment_loss‚1.287490599246389 1loss/SRqqt1/detection_loss‚1.1621776901442429 % _timestamp‚1.7843325319676712e+09 1loss/FBt1/assignment_loss‚0.09425421339628827 1loss/SRqqt2/detection_loss‚0.7325334584497211 /loss/FBt1/detection_loss‚0.5504125513373695 epoch‚3 trainer/global_step‚1549 2loss/SRqqt1/assignment_loss‚0.8727616525299301 -loss/FRt1/detection_loss‚0.63008733588358 /loss/FBt2/detection_loss‚1.3094623336801305 _runtime‚ 2692.55280677~ž–2ˆŸ¢ ‚(:” ûëÒ²öÇ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚147.487' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.497400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M6±«·£ ‚(:¬ ûëÒ¾èØÉ disk.sda.in‚0& disk.nvme0c0n1.out‚18776.724609375 disk.sda3.in‚0 network.recv‚697157 disk.rbd1.in‚0$ memory_percent‚94.05407905578613 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 3364.41015625 disk.dm-0.in‚ 3364.41015625 disk.dm-1.out‚85.3125 disk.sda.out‚85.3125! disk.dm-0.out‚18776.724609375 proc.cpu.threads‚17 disk.rbd1.out‚7.265625* disk./.usagePercent‚39.074918413470925$ disk./.usageGB‚582.2445220947266' proc.memory.availableMB‚ 974.1796875 disk.sda2.in‚0! disk.nvme0n1.in‚ 3364.41015625# proc.memory.rssMB‚ 2846.41796875* proc.memory.percent‚17.373156547546387 disk.rbd0.in‚0$ disk.nvme0n1.out‚18776.724609375 disk.sda1.out‚0 disk.sda3.out‚85.3125 cpu‚25.44856372012596 network.sent‚ 2.776699e+06 disk.rbd0.out‚0g~\”¤ ‚(:‰ Š‚ëÒÌÍÖÈ gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚332.701' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.5402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒÌ&}Å¥ ‚(:º Š‚ëÒºî”Ê disk.sda3.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 3364.41015625# disk.dm-0.out‚18783.16650390625# proc.memory.rssMB‚ 2845.66796875$ disk./.usageGB‚582.2446517944336 disk.dm-1.in‚0 disk.sda.out‚ 85.5859375 disk.drbd2165.out‚0( proc.memory.availableMB‚ 974.44140625) disk./.usagePercent‚39.07492711772709 disk.sda1.in‚0( disk.nvme0c0n1.out‚18783.16650390625! disk.nvme0n1.in‚ 3364.41015625& disk.nvme0n1.out‚18783.16650390625 disk.sda1.out‚0 disk.sda.in‚0 disk.drbd2165.in‚0 cpu‚25.448197510648324 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.out‚ 85.5859375 disk.rbd1.in‚0 disk.rbd1.out‚7.34375 network.recv‚701593 network.sent‚ 2.84183e+06 disk.sda3.out‚ 85.5859375 disk.rbd0.out‚0$ memory_percent‚94.05248165130615 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 3364.41015625* proc.memory.percent‚17.368578910827637’œ¶fž¦ ‚(:“ ™‚ëÒÑèñÆ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚328.649' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.72980000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\øsŽ¶§ ‚(:« ™‚ëÒÇéÈÉ disk.sda1.in‚0 disk.sda2.in‚0# memory_percent‚94.0509557723999 disk.nvme0c0n1.in‚3365.375 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 974.69140625 disk.sda3.out‚ 85.8515625 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2845.66796875 network.sent‚ 2.844581e+06 network.recv‚704663& disk.nvme0n1.out‚18792.23583984375( disk.nvme0c0n1.out‚18792.23583984375 disk.dm-0.in‚3365.375 disk.rbd0.out‚0 disk.dm-1.out‚ 85.8515625 disk.rbd1.out‚7.34375 disk.sda.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda.out‚ 85.8515625) disk./.usagePercent‚39.07493607799079* proc.memory.percent‚17.368578910827637 cpu‚25.447674736552905$ disk./.usageGB‚582.2447853088379# disk.dm-0.out‚18792.23583984375 disk.nvme0n1.in‚3365.375Q„”¨ ‚(:‰ ¨‚ëÒ®©¾È gpu.0.gpu‚88.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚288.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.6878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[>´U¿© ‚(:´ ¨‚ëҧʪÉ) disk./.usagePercent‚39.07489460477024$ disk./.usageGB‚582.2441673278809 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 3416.16796875# disk.dm-0.out‚18802.07373046875 disk.sda2.out‚0 disk.sda.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 3416.16796875 cpu‚25.447330647845718 disk.rbd0.out‚0 network.sent‚ 2.84892e+06& disk.nvme0n1.out‚18802.07373046875 disk.sda3.out‚86.03125 disk.rbd1.in‚0( proc.memory.availableMB‚ 974.91796875! disk.nvme0n1.in‚ 3416.16796875 disk.sda2.in‚0 disk.dm-1.in‚0* proc.memory.percent‚17.373156547546387 disk.dm-1.out‚86.03125 disk.rbd1.out‚7.40625 disk.sda.out‚86.03125 disk.drbd2165.in‚0$ memory_percent‚94.04957294464111 network.recv‚707638 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.in‚0( disk.nvme0c0n1.out‚18802.07373046875# proc.memory.rssMB‚ 2846.41796875n&”ª ‚(:‰ ·‚ëÒÒÁ©Ç gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚337.146' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.4292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01±vÄÈ« ‚(:½ ·‚ëÒþÍ°É disk.sda2.in‚0 disk.drbd2165.in‚0* proc.memory.percent‚17.373156547546387 disk.sda.in‚0 disk.dm-0.in‚ 3416.16796875& disk.nvme0n1.out‚18805.34521484375 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚711278# disk.nvme0c0n1.in‚ 3416.16796875 disk.sda1.out‚0 disk.dm-1.out‚ 86.27734375( disk.nvme0c0n1.out‚18805.34521484375 disk.sda3.out‚ 86.27734375# proc.memory.rssMB‚ 2846.41796875 network.sent‚ 2.853741e+06 disk.rbd1.in‚0 cpu‚25.44689523643477) disk./.usagePercent‚39.07489255670997 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚7.40625 disk.drbd2165.out‚0# disk.dm-0.out‚18805.34521484375 disk.rbd0.in‚0$ disk./.usageGB‚582.2441368103027 disk.sda.out‚ 86.27734375! disk.nvme0n1.in‚ 3416.16796875$ memory_percent‚94.05105113983154( proc.memory.availableMB‚ 974.67578125ß³I\2¬ - _wandbruntime‚2774  _runtime‚2774.ýÑÇ­ Á ' lr-AdamW/pg1‚0.00047803217660922896 ' lr-AdamW/pg2‚0.00047803217660922896 trainer/global_step‚1599 # _timestamp‚1.78433261387539e+09 _runtime‚2774.460764287 _step‚127™õЗË® Å ' lr-AdamW/pg1‚0.00047803217660922896 ' lr-AdamW/pg2‚0.00047803217660922896 trainer/global_step‚1599 # _timestamp‚1.78433261387539e+09 _runtime‚2774.460764287 _step‚127«ÄÐ2¯ - _wandbruntime‚2774  _runtime‚2774@ÔAQî° çO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13797295 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4513538 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚456372614725632 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8613544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4099753 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13529983 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1497714393586 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1498932737536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚456372614725632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4513786 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1936345834496 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457872892852736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13529989 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13807268 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27337257 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚457872892852736 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126251054314496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚457671430076656 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚124314708480000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚456175060283392 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8613291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13807268 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13529989 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚456372588552192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚456372588552192 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMRÉSÿonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843326138762817e+09 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457871521289728 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457871521289728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1498932737536 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1500278127104 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126250651686912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27327278 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1500278127104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1599 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13797295 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4099758 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1496369793264 _runtime‚2774.461584376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚457672800104946 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚457872892852736 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126251054314496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚457671430076656 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚124314708480000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚456175060283392 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8613291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13807268 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13529989 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚456372588552192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚456372588552192 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843326138762817e+09 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457871521289728 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457871521289728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1498932737536 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1500278127104 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126250651686912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27327278 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1500278127104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1599 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13797295 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4099758 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1496369793264 _runtime‚2774.461584376€}¬è<Ÿ² ‚(:” Æ‚ëÒ…ý¿È gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚200.917' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚40.183400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KoMÁ³ ‚(:¶ Æ‚ëÒ®É¥É disk.sda1.out‚0 disk.sda3.out‚ 86.8984375 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 cpu‚25.44655650378449 network.recv‚714391 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 3417.1328125 disk.nvme0n1.in‚ 3417.1328125# proc.memory.rssMB‚ 2846.41796875 network.sent‚ 2.85871e+06 disk.dm-1.in‚0 disk.sda.in‚0" disk.nvme0c0n1.in‚ 3417.1328125 disk.sda1.in‚0) disk./.usagePercent‚39.07494299019422 disk.dm-1.out‚ 86.8984375 disk.rbd1.out‚7.484375 disk.drbd2165.in‚0' proc.memory.availableMB‚ 974.4296875* proc.memory.percent‚17.373156547546387$ disk./.usageGB‚582.2448883056641( disk.nvme0c0n1.out‚18814.37646484375# disk.dm-0.out‚18814.37646484375$ memory_percent‚94.05255317687988 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda.out‚ 86.8984375& disk.nvme0n1.out‚18814.37646484375l qw2´ - _wandbruntime‚2775  _runtime‚2775Õˆ%÷&µ ñM D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1498637859944 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8618814 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚456452967489536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13538195 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27353955 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458150447055360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 _runtime‚2775.867805987 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8618577 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124392194887680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13815760 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1497293259510 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458151818625536 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1499857773056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚129 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27353955 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13538189 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13538189 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1499857773056 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124391826796544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1937515474944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚457951605349480 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚456650589282304 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚456650615455744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1937550004224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13805773 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8618577 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124392194887680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13815760 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1497293259510 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458151818625536 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1499857773056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚129 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27353955 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13538189 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13538189 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1499857773056 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124391826796544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1937515474944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚457951605349480 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚456650589282304 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚456650615455744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1937550004224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13805773 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E½ “| & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 trainer/global_step‚1599 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_1_of_1‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11531693488359451 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16344696283340454 /particle/accuracy_4_of_4‚0.9363824129104614 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7336517572402954 +particle/specificity‚0.9055944085121155 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6375258564949036 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _runtime‚2804.098148992 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.8436105847358704 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955057501792908 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6351382732391357 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7317460179328918 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5629376769065857 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9959455132484436 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.684618353843689 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6932879686355591 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11816712468862534 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6939926743507385 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34416040778160095 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5863602161407471 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7413076758384705 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7332043647766113 )jet/accuracy_2_of_4‚0.766822338104248 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9969112873077393 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.783965528011322 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3293662667274475 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1496400386095047 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915412068367004 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9984639286994934 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4410061240196228 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6314101815223694 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4004629850387573 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9253424406051636 ,jet/accuracy_4_of_4‚0.012732365168631077 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1496332436800003 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5595654249191284 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5867871046066284 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.44793882966041565 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03404386341571808 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 *loss/val_total_loss‚11.262685352005217 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5833941698074341 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6272486448287964 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.562385082244873 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8555823564529419 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2653401792049408 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15136006474494934 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013302232138812542 ,jet/accuracy_3_of_3‚0.058563582599163055 jet/accuracy_2_of_5‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.47995299100875854 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.783965528011322 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9940875768661499 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13317658007144928 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6931639909744263 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.3125 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1624416559934616 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9973902106285095 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.35728180408477783 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02789902687072754 +jet/accuracy_3_of_4‚0.06812838464975357 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.694617748260498 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014373227022588253 particle/accuracy_2_of_4‚1 jet/accuracy_3_of_5‚1 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.996673047542572 *jet/accuracy_2_of_3‚0.4975034296512604 'particle/f_score‚0.6288450360298157 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.37425410747528076 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9878305792808533 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14000001549720764 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8459078669548035 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2185056507587433 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3432298004627228 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.996673047542572 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9830824732780457 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.443396657705307 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5798043012619019 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.024979110807180405 +particle/sensitivity‚0.6169621348381042 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17653128504753113 *jet/accuracy_1_of_4‚0.9541816711425781 /particle/accuracy_3_of_3‚0.9989582896232605 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1533956080675125 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08554339408874512 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4375000298023224 jet/accuracy_4_of_5‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12681972980499268 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.994875431060791 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1531306505203247 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6330668330192566 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1606287956237793 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08729156851768494 /particle/accuracy_3_of_4‚0.9915816187858582 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9959985017776489 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9051416516304016 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9920982122421265 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9934089183807373 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5995261073112488 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6900035738945007 jet/accuracy_5_of_5‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 +jet/accuracy_1_of_1‚0.28453242778778076 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6539983749389648 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5867871046066284 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08489172160625458 particle/accuracy_1_of_3‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6916369199752808 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1310819387435913 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14542093873023987 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14689630270004272 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7406274080276489 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15655824542045593 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5841078758239746 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9973545074462891 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5911458730697632 epoch‚3 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930034875869751 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 jet/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6493998169898987 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08729156851768494 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9947485327720642 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6145163774490356 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22802141308784485 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9958407282829285 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5087065100669861 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7363945841789246 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8411409258842468 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19353319704532623 *jet/accuracy_1_of_2‚0.7504265904426575 *jet/accuracy_1_of_3‚0.8964186310768127 particle/accuracy_3_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9890558123588562 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5118940472602844 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6888236999511719 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971046447753906 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1421995311975479 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5608693361282349 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12386664748191833 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14080879092216492 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007143791764974594 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14059431850910187 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08495956659317017 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 /particle/accuracy_2_of_2‚0.9998962879180908 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7843326435121715e+09 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938080310821533 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6936423182487488 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14001105725765228 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15210767090320587 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_3‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.12213045358657837 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.620770275592804 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5876529812812805 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.49958333373069763 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7361111044883728 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.23818440735340118 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5849917531013489 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5763843059539795 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15136006474494934 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6937007904052734 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_4_of_5‚0 _step‚131 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 7validation_average_jet_accuracy‚0.33031225204467773 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6680172681808472 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0837770402431488 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_4‚1 particle/accuracy_5_of_5‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3388616740703583 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6858673095703125 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7361111044883728Vß’:(¾ ˜| @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_4_of_5‚0 _step‚131 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 7validation_average_jet_accuracy‚0.33031225204467773 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6680172681808472 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0837770402431488 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_4‚1 particle/accuracy_5_of_5‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3388616740703583 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6858673095703125 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7361111044883728 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 trainer/global_step‚1599 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_1_of_1‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11531693488359451 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16344696283340454 /particle/accuracy_4_of_4‚0.9363824129104614 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7336517572402954 +particle/specificity‚0.9055944085121155 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6375258564949036 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _runtime‚2804.098148992 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.8436105847358704 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955057501792908 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6351382732391357 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7317460179328918 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5629376769065857 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9959455132484436 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.684618353843689 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6932879686355591 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11816712468862534 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6939926743507385 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34416040778160095 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5863602161407471 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7413076758384705 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7332043647766113 )jet/accuracy_2_of_4‚0.766822338104248 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9969112873077393 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.783965528011322 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3293662667274475 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1496400386095047 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915412068367004 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9984639286994934 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4410061240196228 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6314101815223694 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4004629850387573 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9253424406051636 ,jet/accuracy_4_of_4‚0.012732365168631077 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1496332436800003 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5595654249191284 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5867871046066284 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.44793882966041565 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03404386341571808 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 *loss/val_total_loss‚11.262685352005217 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5833941698074341 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6272486448287964 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.562385082244873 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8555823564529419 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2653401792049408 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15136006474494934 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013302232138812542 ,jet/accuracy_3_of_3‚0.058563582599163055 jet/accuracy_2_of_5‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.47995299100875854 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.783965528011322 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9940875768661499 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13317658007144928 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6931639909744263 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.3125 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1624416559934616 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9973902106285095 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.35728180408477783 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02789902687072754 +jet/accuracy_3_of_4‚0.06812838464975357 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.694617748260498 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014373227022588253 particle/accuracy_2_of_4‚1 jet/accuracy_3_of_5‚1 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.996673047542572 *jet/accuracy_2_of_3‚0.4975034296512604 'particle/f_score‚0.6288450360298157 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.37425410747528076 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9878305792808533 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14000001549720764 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8459078669548035 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2185056507587433 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3432298004627228 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.996673047542572 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9830824732780457 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.443396657705307 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5798043012619019 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.024979110807180405 +particle/sensitivity‚0.6169621348381042 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17653128504753113 *jet/accuracy_1_of_4‚0.9541816711425781 /particle/accuracy_3_of_3‚0.9989582896232605 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1533956080675125 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08554339408874512 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4375000298023224 jet/accuracy_4_of_5‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12681972980499268 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.994875431060791 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1531306505203247 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6330668330192566 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1606287956237793 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08729156851768494 /particle/accuracy_3_of_4‚0.9915816187858582 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9959985017776489 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9051416516304016 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9920982122421265 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9934089183ʸŒä807373 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5995261073112488 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6900035738945007 jet/accuracy_5_of_5‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 +jet/accuracy_1_of_1‚0.28453242778778076 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6539983749389648 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5867871046066284 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08489172160625458 particle/accuracy_1_of_3‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6916369199752808 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1310819387435913 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14542093873023987 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14689630270004272 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7406274080276489 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15655824542045593 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5841078758239746 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9973545074462891 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5911458730697632 epoch‚3 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930034875869751 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 jet/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6493998169898987 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08729156851768494 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9947485327720642 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6145163774490356 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22802141308784485 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9958407282829285 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5087065100669861 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7363945841789246 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8411409258842468 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19353319704532623 *jet/accuracy_1_of_2‚0.7504265904426575 *jet/accuracy_1_of_3‚0.8964186310768127 particle/accuracy_3_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9890558123588562 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5118940472602844 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6888236999511719 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971046447753906 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1421995311975479 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5608693361282349 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12386664748191833 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14080879092216492 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007143791764974594 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14059431850910187 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08495956659317017 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 /particle/accuracy_2_of_2‚0.9998962879180908 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7843326435121715e+09 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938080310821533 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6936423182487488 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14001105725765228 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15210767090320587 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_3‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.12213045358657837 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.620770275592804 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5876529812812805 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.49958333373069763 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7361111044883728 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.23818440735340118 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5849917531013489 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5763843059539795 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15136006474494934 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6937007904052734 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1ƒ •‘Ä“¿ ‚(:ˆ ä‚ëÒ¶ðÞÊ gpu.0.gpu‚26.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µn©ÉÀ ‚(:¾ ä‚ëÒþ½àË disk.rbd0.in‚0 cpu‚25.470661402487323# disk./.usageGB‚583.042537689209$ memory_percent‚96.60265445709229( proc.memory.availableMB‚ 556.62109375 disk.dm-0.in‚ 3458.26953125 disk.sda3.in‚0 disk.sda3.out‚ 87.39453125 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda.in‚0# disk.dm-0.out‚18828.86865234375& disk.nvme0n1.out‚18828.86865234375 disk.dm-1.out‚ 87.39453125( disk.nvme0c0n1.out‚18828.86865234375 disk.rbd1.in‚0 disk.rbd1.out‚7.546875 network.sent‚ 2.963491e+06 disk.rbd0.out‚0 disk.sda2.out‚0* proc.memory.percent‚19.928765296936035 network.recv‚723299 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 3458.26953125 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 3458.26953125 disk.sda1.in‚0# proc.memory.rssMB‚ 3265.12890625 proc.cpu.threads‚17) disk./.usagePercent‚39.12847390959632 disk.sda.out‚ 87.39453125 disk.sda1.out‚0NMšO’Á ‚(:‡ ó‚ëÒéóØÇ gpu.0.gpu‚73.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚307.24' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚61.448 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡jyÇ ‚(:¼ ó‚ëÒèâÜÈ disk.sda2.out‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 3459.2578125( proc.memory.availableMB‚ 1986.0859375$ disk./.usageGB‚583.0118408203125 disk.sda3.out‚ 87.66015625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚21401.31396484375 network.recv‚727678 disk.nvme0n1.in‚ 3459.2578125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 87.66015625 disk.sda.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 3459.2578125# proc.memory.rssMB‚ 2847.73828125* disk./.usagePercent‚39.126413816968295& disk.nvme0n1.out‚21401.31396484375 disk.rbd1.out‚7.546875) proc.memory.percent‚17.38121509552002 network.sent‚ 3.020904e+06 disk.sda3.in‚0 cpu‚25.466128039103037 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 87.66015625# disk.dm-0.out‚21364.43896484375$ memory_percent‚87.87789344787598 disk.sda1.out‚0µ O’à ‚(:‡ ‚ƒëÒ¢ñ±Ç gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=©¹xÆÄ ‚(:» ‚ƒëÒÕÒÚÊ network.sent‚ 3.025945e+06$ disk./.usageGB‚582.2983322143555# disk.dm-0.out‚21596.48291015625 disk.rbd0.in‚0 disk.dm-0.in‚ 3460.16015625& disk.nvme0n1.out‚21596.48291015625 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda.out‚ 88.01171875! disk.nvme0n1.in‚ 3460.16015625 disk.sda3.in‚0 disk.sda3.out‚ 88.01171875 disk.rbd0.out‚0# proc.memory.rssMB‚ 2847.73828125 disk.sda1.in‚0# memory_percent‚93.8119649887085 network.recv‚731053 disk.sda.in‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 3460.16015625) proc.memory.percent‚17.38121509552002 disk.sda2.in‚0 disk.dm-1.out‚ 88.01171875 disk.rbd1.in‚0) proc.memory.availableMB‚ 1013.84765625 cpu‚25.465832033833163 proc.cpu.threads‚17) disk./.usagePercent‚39.07852965574892 disk.rbd1.out‚7.625( disk.nvme0c0n1.out‚21596.48291015625é—`œÅ ‚(:‘ ‘ƒëÒù“°È gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚330.528' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.10560000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û/Ï ÀÆ ‚(:µ ‘ƒëÒÁ™÷É& proc.memory.availableMB‚ 1013.84375 disk.sda.out‚ 88.33984375 disk.drbd2165.out‚0 disk.sda1.out‚0) proc.memory.percent‚17.38121509552002 disk.sda3.in‚0 proc.cpu.threads‚17' disk./.usagePercent‚39.078534007877 disk.sda2.out‚0 disk.rbd1.out‚7.625# disk.dm-0.out‚21601.72607421875& disk.nvme0n1.out‚21601.72607421875 disk.sda1.in‚0( disk.nvme0c0n1.out‚21601.72607421875 disk.sda3.out‚ 88.33984375# memory_percent‚93.8119888305664 cpu‚25.465328752011533 network.recv‚734566# disk./.usageGB‚582.298397064209 disk.dm-1.out‚ 88.33984375 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 3460.16015625# proc.memory.rssMB‚ 2847.73828125 network.sent‚ 3.030772e+06 disk.sda2.in‚0 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 3460.16015625! disk.nvme0n1.in‚ 3460.16015625 disk.rbd0.out‚0·àçú”Ç ‚(:‰  ƒëÒ•‹ÖÇ gpu.0.gpu‚50.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û;SúÁÈ ‚(:¶  ƒëÒ›°äÉ! disk.dm-0.out‚21609.318359375 disk.sda.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.46481951728755 proc.cpu.threads‚17& disk.nvme0c0n1.out‚21609.318359375 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 3505.09765625 disk.rbd1.out‚7.6875( proc.memory.availableMB‚ 1013.5546875 disk.sda3.out‚ 88.59765625$ memory_percent‚93.81375312805176 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 3505.09765625! disk.nvme0n1.in‚ 3505.09765625* disk./.usagePercent‚39.076728642745536 disk.sda.out‚ 88.59765625 disk.rbd0.out‚0$ disk./.usageGB‚582.2714958190918# proc.memory.rssMB‚ 2847.73828125 network.sent‚ 3.035746e+06 network.recv‚737666 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 88.59765625) proc.memory.percent‚17.38121509552002$ disk.nvme0n1.out‚21609.318359375¦ëîÉ ‚(:’ ¯ƒëÒÕÚÌÇ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚295.159' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.031800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nÁ?îÆÊ ‚(:» ¯ƒëÒøÙÒÉ disk.sda2.out‚0 disk.rbd1.out‚7.6875 disk.drbd2165.out‚0 disk.dm-0.in‚ 3505.09765625 network.sent‚ 3.040724e+06# disk./.usageGB‚582.271411895752 disk.sda2.in‚0 disk.dm-1.out‚ 88.77734375# disk.nvme0c0n1.in‚ 3505.09765625# proc.memory.rssMB‚ 2847.01953125 disk.rbd0.in‚0# memory_percent‚93.8103437423706 disk.dm-1.in‚0( disk.nvme0c0n1.out‚21614.78955078125# disk.dm-0.out‚21614.78955078125& disk.nvme0n1.out‚21614.78955078125 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.sda1.in‚0) proc.memory.percent‚17.37682819366455 proc.cpu.threads‚17 disk.rbd0.out‚0) proc.memory.availableMB‚ 1014.11328125 network.recv‚741188 disk.sda.in‚0! disk.nvme0n1.in‚ 3505.09765625) disk./.usagePercent‚39.07672301057979 disk.sda.out‚ 88.77734375 disk.sda3.out‚ 88.77734375 cpu‚25.464315159287757{í+b2Ë - _wandbruntime‚2891  _runtime‚2891X3è§ÇÌ Á & lr-AdamW/pg2‚0.0004776553065468489 trainer/global_step‚1649 % _timestamp‚1.7843327311166434e+09 _runtime‚2891.701931044 _step‚132 & lr-AdamW/pg1‚0.0004776553065468489Sº¦¨ÌÍ Æ & lr-AdamW/pg1‚0.0004776553065468489 & lr-AdamW/pg2‚0.0004776553065468489 trainer/global_step‚1649 % _timestamp‚1.7843327311166434e+09 _runtime‚2891.701931044 _step‚132„çt€½2Î - _wandbruntime‚2891  _runtime‚2891|£üUì'Ï æO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1549780638136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8921681 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1552436672000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8921925 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚474100606492672 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4686099 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14282034 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4235821 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚130240386271232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚475444864190904 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28364302 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚474100606492672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚474100632666112 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4685860 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1548436037814 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1551091282432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _step‚133 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28354323 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚475651697775104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚474100632666112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1552436672000 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚473895058124800 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14072289 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚475651697775104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14072295 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚1649 _runtime‚2891.704103526 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚475443494162614 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1551091282432 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14072289 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚475651697775104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14072295 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚1649 _runtime‚2891.704103526 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚475443494162614 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1551091282432 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8921681 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1552436672000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8921925 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚474100606492672 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4686099 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14282034 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4235821 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚130240386271232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚475444864190904 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28364302 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚474100606492672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚474100632666112 SGDeviceStatsMonitor.on_t¯c)þ^rain_batch_start/inactive_split.small_pool.freed‚4685860 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1548436037814 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1551091282432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _step‚133 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28354323 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚475651697775104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚474100632666112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1552436672000 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚473895058124800 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51…ßrº2Ñ - _wandbruntime‚2893  _runtime‚2893W6;Mö&Ò ðM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚474378607222784 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14300499 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1552016317952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4238364 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚475722299407036 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚474172965330944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 trainer/global_step‚1649 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4688851 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1553361714688 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14080495 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130319079403520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28381000 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚474378607222784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚2893.113387956 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14290512 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8927220 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1552016317952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28381000 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚130319479926784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28371007 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8926984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128314259427328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2004852408320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1550704104494 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14080501 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚474378633396224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1549359504060 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚134 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚475931995110912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14290512 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚474172939902976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14080501 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28371007 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1553361714688 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚475930623540736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚474378633396224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4238369 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784332732527872e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987£YÂ3û&Ó õM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14290512 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8927220 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1552016317952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28381000 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚130319479926784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28371007 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8926984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128314259427328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2004852408320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1550704104494 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14080501 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚474378633396224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1549359504060 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚134 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚475931995110912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14290512 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚474172939902976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14080501 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28371007 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1553361714688 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚475930623540736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚474378633396224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4238369 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784332732527872e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚474378607222784 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14300499 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1552016317952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4238364 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚475722299407036 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚474172965330944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 trainer/global_step‚1649 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4688851 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1553361714688 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14080495 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130319079403520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28381000 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚474378607222784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚2893.113387956 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176†CŠÙe2Ô - _wandbruntime‚2893  _runtime‚2893%C`±Õ ÿ $ _timestamp‚1.784332732607455e+09 _step‚135 0loss/FBt2/assignment_loss‚0.1352215664737476 /loss/FBt2/detection_loss‚1.2310783154020706 0loss/FRt1/assignment_loss‚2.2499421183577555 0loss/FRt2/assignment_loss‚1.2160612513871705 1loss/FBt1/assignment_loss‚0.08752834849132726 0loss/SRqqt1/detection_loss‚1.276080523437562 1loss/SRqqt2/detection_loss‚0.7553451352816108 trainer/global_step‚1649 _runtime‚2893.192691929 2loss/SRqqt1/assignment_loss‚0.8603883731509598 2loss/SRqqt2/assignment_loss‚0.7214405948195092 /loss/FRt2/detection_loss‚1.7423640553982294 /loss/FBt1/detection_loss‚0.4741052215203842 epoch‚4 /loss/FRt1/detection_loss‚0.6150310764770581 &loss/total_loss‚11.364586580197384›k"ñ Ö „ 0loss/FBt2/assignment_loss‚0.1352215664737476 /loss/FBt2/detection_loss‚1.2310783154020706 0loss/FRt1/assignment_loss‚2.2499421183577555 0loss/FRt2/assignment_loss‚1.2160612513871705 1loss/FBt1/assignment_loss‚0.08752834849132726 0loss/SRqqt1/detection_loss‚1.276080523437562 1loss/SRqqt2/detection_loss‚0.7553451352816108 trainer/global_step‚1649 _runtime‚2893.192691929 2loss/SRqqt1/assignment_loss‚0.8603883731509598 2loss/SRqqt2/assignment_loss‚0.7214405948195092 /loss/FRt2/detection_loss‚1.7423640553982294 /loss/FBt1/detection_loss‚0.4741052215203842 epoch‚4 /loss/FRt1/detection_loss‚0.6150310764770581 &loss/total_loss‚11.364586580197384 $ _timestamp‚1.784332732607455e+09 _step‚135‡zÍvž× ‚(:“ ¾ƒëÒ®‘ÖÉ gpu.0.gpu‚86.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚333.524' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.70479999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ͤÅØ ‚(:º ¾ƒëÒïüîÊ network.sent‚ 3.045754e+06 disk.dm-1.out‚ 89.18359375( disk.nvme0c0n1.out‚21624.20556640625$ disk./.usageGB‚582.2719879150391 disk.rbd0.in‚0 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 3506.390625# disk.dm-0.out‚21624.20556640625 disk.sda3.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2847.01953125 disk.sda.out‚ 89.18359375 disk.dm-0.in‚ 3506.390625) proc.memory.percent‚17.37682819366455* disk./.usagePercent‚39.076761667717456 disk.nvme0n1.in‚ 3506.390625 disk.rbd0.out‚0 disk.sda2.out‚0& disk.nvme0n1.out‚21624.20556640625 cpu‚25.463962672537765 disk.sda1.in‚0 disk.rbd1.out‚7.765625 disk.drbd2165.in‚0$ memory_percent‚93.81043910980225) proc.memory.availableMB‚ 1014.09765625 proc.cpu.threads‚17 network.recv‚744760 disk.sda1.out‚0 disk.sda3.out‚ 89.18359375 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.in‚0·Ëöæ”Ù ‚(:‰ ̓ëÒ÷βÉ gpu.0.gpu‚67.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚112.708' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5416 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QΓŠÀÚ ‚(:µ ̓ëÒò¬åÊ disk.sda3.in‚0 disk.dm-0.in‚ 3507.3671875& disk.nvme0n1.out‚21646.92626953125$ memory_percent‚93.81229877471924 disk.sda3.out‚ 91.171875 disk.dm-1.out‚ 91.171875 disk.rbd1.in‚0 cpu‚25.46338195709959 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 3507.3671875( disk.nvme0c0n1.out‚21646.92626953125$ disk./.usageGB‚582.2764930725098 disk.rbd1.out‚7.765625) proc.memory.availableMB‚ 1013.79296875 network.sent‚ 3.110804e+06* disk./.usagePercent‚39.077064012615395 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0# disk.dm-0.out‚21646.92626953125 disk.nvme0n1.in‚ 3507.3671875 disk.sda1.in‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 2847.7421875) proc.memory.percent‚17.38123893737793 proc.cpu.threads‚17 network.recv‚749345 disk.sda.out‚ 91.171875‹Ö”Û ‚(:‰ ܃ëÒ³‚ŒÊ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚292.207' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.4414 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Yß4¿Ü ‚(:´ ܃ëÒ­‡ÂË network.recv‚752652 disk.dm-1.out‚ 93.2734375 disk.dm-0.in‚ 3637.35546875! disk.nvme0n1.in‚ 3637.35546875 disk.sda3.out‚ 93.2734375 cpu‚25.462792384532086$ disk./.usageGB‚582.2717170715332 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 93.2734375 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd0.out‚0$ memory_percent‚93.81685256958008 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 3637.35546875! disk.dm-0.out‚21656.033203125 disk.sda1.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 1013.046875" proc.memory.rssMB‚ 2847.7421875 disk.rbd1.out‚7.828125$ disk.nvme0n1.out‚21656.033203125 proc.cpu.threads‚17 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚21656.033203125 network.sent‚ 3.115732e+06* disk./.usagePercent‚39.076743491182526 disk.sda.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0) proc.memory.percent‚17.38123893737793Z2v`žÝ ‚(:“ ëƒëÒ¬ðÊÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.135' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.82699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑýLÒÉÞ ‚(:¾ ëƒëÒ£ÑàÈ) disk./.usagePercent‚39.07674195513732# disk.nvme0c0n1.in‚ 3724.80078125 disk.rbd0.out‚0 disk.dm-1.in‚0! disk.nvme0n1.in‚ 3724.80078125& disk.nvme0n1.out‚21667.61865234375 network.sent‚ 3.118696e+06 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚21667.61865234375 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.37685203552246) proc.memory.availableMB‚ 1014.04296875 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2847.0234375 disk.dm-0.in‚ 3724.80078125 disk.sda2.in‚0$ memory_percent‚93.81077289581299$ disk./.usageGB‚582.2716941833496 disk.sda.in‚0 disk.sda3.out‚ 93.61328125 disk.sda2.out‚0 disk.dm-1.out‚ 93.61328125 network.recv‚756083 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 93.61328125# disk.dm-0.out‚21667.61865234375 disk.rbd1.out‚7.828125 cpu‚25.462355324788636 proc.cpu.threads‚17cå”ß ‚(:‰ úƒëÒ÷ŸòÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò7Ê:Áà ‚(:¶ úƒëÒ˜ªíÉ disk.dm-1.in‚0 disk.rbd0.out‚0 network.recv‚759287! disk.nvme0c0n1.in‚ 3725.765625( disk.nvme0c0n1.out‚21674.15185546875 disk.rbd0.in‚0" proc.memory.rssMB‚ 2847.0234375 proc.cpu.threads‚17) disk./.usagePercent‚39.07676320376267 disk.dm-1.out‚ 93.90234375& disk.nvme0n1.out‚21674.15185546875 disk.sda3.out‚ 93.90234375# memory_percent‚93.8107967376709( proc.memory.availableMB‚ 1014.0390625 disk.sda2.out‚0 disk.dm-0.in‚ 3725.765625 disk.nvme0n1.in‚ 3725.765625 disk.sda2.in‚0 disk.sda.out‚ 93.90234375) proc.memory.percent‚17.37685203552246 cpu‚25.461854026214755 network.sent‚ 3.123765e+06 disk.sda.in‚0# disk.dm-0.out‚21674.15185546875 disk.sda1.out‚0$ disk./.usageGB‚582.2720108032227 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚0 disk.rbd1.out‚7.890625 disk.drbd2165.out‚0 disk.sda1.in‚0l^-”á ‚(:‰ ‰„ëÒÇæÊÇ gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚111.883' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3766 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VŠÈœ×â ‚(:Ì ‰„ëÒùÎÉ network.sent‚ 3.12871e+06& disk.nvme0n1.out‚21678.77880859375 disk.sda3.in‚ 0.47265625$ memory_percent‚93.81542205810547 network.recv‚762598 disk.rbd0.in‚0& proc.memory.availableMB‚ 1013.28125 cpu‚25.461388372728667 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625 disk.sda.out‚ 94.7734375! disk.nvme0c0n1.in‚ 3725.765625 disk.dm-0.in‚ 3725.765625 disk.sda1.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚582.2719612121582 disk.dm-1.out‚ 94.7734375 disk.rbd1.out‚7.890625 disk.nvme0n1.in‚ 3725.765625 disk.sda2.in‚0( disk.nvme0c0n1.out‚21678.77880859375 disk.sda2.out‚0" proc.memory.rssMB‚ 2847.7421875) proc.memory.percent‚17.38123893737793 disk.rbd0.out‚0# disk.dm-0.out‚21678.77880859375 disk.sda1.in‚0) disk./.usagePercent‚39.07675987566472 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 94.7734375W>ä;2ã - _wandbruntime‚2975  _runtime‚2975O”k°Çä Á _runtime‚ 2975.0027608 _step‚136 ' lr-AdamW/pg1‚0.00047726325175903426 ' lr-AdamW/pg2‚0.00047726325175903426 trainer/global_step‚1699 % _timestamp‚1.7843328144175732e+09Sê%®Ìå Æ _runtime‚ 2975.0027608 _step‚136 ' lr-AdamW/pg1‚0.00047726325175903426 ' lr-AdamW/pg2‚0.00047726325175903426 trainer/global_step‚1699 % _timestamp‚1.7843328144175732e+09ˆ˧Oä2æ - _wandbruntime‚2975  _runtime‚2975©œxnç çO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requ*£'ested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚487789172461568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14717759 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚1699 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1597407213056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4824539 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4362976 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1598752602624 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29200354 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134174842042368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14482595 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚137 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14482595 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9187510 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚487789147033600 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚488000669171712 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 % _timestamp‚1.7843328144189508e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9187770 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4824794 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚488000642998272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4362971 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚489598050211328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚488000642998272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚132111093403648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚489385190212812 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚489599421774336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2063748638720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚489598050211328 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29190375 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚134175242572800 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14482589 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14482589 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14717759 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2063781078016R~‚Nò'è ìO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14482589 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14482589 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14717759 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2063781078016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚487789172461568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14717759 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚1699 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1597407213056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4824539 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4362976 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1598752602624 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29200354 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134174842042368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14482595 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚137 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14482595 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9187510 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚487789147033600 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚488000669171712 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 % _timestamp‚1.7843328144189508e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9187770 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4824794 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚488000642998272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4362971 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚489598050211328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚488000642998272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚132111093403648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚489385190212812 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚489599421774336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2063748638720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚489598050211328 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29190375 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚134175242572800 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0‰^bi2é - _wandbruntime‚2976  _runtime‚29762t5÷&ê ñM RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚488067028811776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚489876975976960 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 % _timestamp‚1.7843328158567903e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1596941217602 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29207059 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4365519 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14716264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4827521 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚132188947902464 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚488278669901824 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1599677645312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9192794 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29217052 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14490795 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14490795 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚132188579811328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1598332248576 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14726251 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9193040 _step‚138 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4365514 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14716264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134253527138304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2064979759104 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚1699 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2064947326976 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1595596617168 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚489878347547136¶Â¨ÞC ë öM EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1599677645312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9192794 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29217052 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14490795 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14490795 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚132188579811328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1598332248576 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14726251 NBDeviceStatsMonitor.on_train_batch_end/inÔÅ™~¹active_split.all.allocated‚9193040 _step‚138 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4365514 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14716264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134253527138304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2064979759104 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚1699 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2064947326976 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1595596617168 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚489878347547136 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚488067028811776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚489876975976960 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 % _timestamp‚1.7843328158567903e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1596941217602 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29207059 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4365519 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14716264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4827521 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚132188947902464 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚488278669901824ŠŽɶ2ì - _wandbruntime‚2976  _runtime‚2976Wž}í € .loss/FRt2/detection_loss‚1.542378794104775 /loss/FBt1/detection_loss‚0.7566465146170692 % _timestamp‚1.7843328159162526e+09 0loss/FRt2/assignment_loss‚1.0544172253589201 2loss/SRqqt2/assignment_loss‚0.6549681082956335 1loss/FBt1/assignment_loss‚0.05629340677969384 0loss/SRqqt1/detection_loss‚1.189132610150488 trainer/global_step‚1699 0loss/FRt1/assignment_loss‚2.2619049168647605 2loss/SRqqt1/assignment_loss‚0.7453250691931679 1loss/SRqqt2/detection_loss‚0.6914545068590697 epoch‚4 _runtime‚2976.501305769 _step‚139 /loss/FBt2/detection_loss‚0.8549797750110256 &loss/total_loss‚10.528515390668275 1loss/FBt2/assignment_loss‚0.14118538951429924 /loss/FRt1/detection_loss‚0.5798290739193728ƒ[ î … % _timestamp‚1.7843328159162526e+09 0loss/FRt2/assignment_loss‚1.0544172253589201 2loss/SRqqt2/assignment_loss‚0.6549681082956335 1loss/FBt1/assignment_loss‚0.05629340677969384 0loss/SRqqt1/detection_loss‚1.189132610150488 trainer/global_step‚1699 0loss/FRt1/assignment_loss‚2.2619049168647605 2loss/SRqqt1/assignment_loss‚0.7453250691931679 1loss/SRqqt2/detection_loss‚0.6914545068590697 epoch‚4 _runtime‚2976.501305769 _step‚139 /loss/FBt2/detection_loss‚0.8549797750110256 &loss/total_loss‚10.528515390668275 1loss/FBt2/assignment_loss‚0.14118538951429924 /loss/FRt1/detection_loss‚0.5798290739193728 .loss/FRt2/detection_loss‚1.542378794104775 /loss/FBt1/detection_loss‚0.7566465146170692‹“Ž,4Ÿï ‚(:” ˜„ëÒßÕ¹Ç gpu.0.gpu‚71.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àyøUáð ‚(:Ö ˜„ëÒ䌗É disk.dm-1.in‚ 0.47265625 cpu‚25.461000341468264 disk.sda.in‚ 0.47265625 disk.sda2.in‚0 disk.rbd1.out‚8.03125% disk.nvme0n1.out‚21937.0595703125 disk.sda1.out‚0 disk.rbd1.in‚0$ memory_percent‚93.81475448608398 network.recv‚765792 disk.sda.out‚ 100.82421875 proc.cpu.threads‚17 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 3726.61328125 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 100.82421875 disk.sda2.out‚0) disk./.usagePercent‚39.10675346236878" disk.dm-0.out‚21937.0595703125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 100.82421875" proc.memory.rssMB‚ 2847.7421875$ disk./.usageGB‚582.7188873291016# disk.nvme0c0n1.in‚ 3726.61328125 disk.rbd0.out‚0) proc.memory.percent‚17.38123893737793' proc.memory.availableMB‚ 1013.390625 network.sent‚ 3.133681e+06 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚21937.0595703125 disk.dm-0.in‚ 3726.61328125ë ïÕžñ ‚(:“ §„ëÒÉ“›Ê gpu.0.gpu‚16.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nÒ–Läò ‚(:Ù §„ëÒ™ÜÌ disk.drbd2165.out‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.47265625 cpu‚25.460478993368145 proc.cpu.threads‚17 network.sent‚ 3.196828e+06# disk.dm-0.out‚22336.44384765625 disk.sda3.in‚ 0.47265625 disk.sda2.out‚0 disk.sda.in‚ 0.47265625" disk.nvme0c0n1.in‚ 3771.5859375) proc.memory.availableMB‚ 1013.64453125$ disk./.usageGB‚582.8560981750488 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.38123893737793 disk.rbd1.out‚8.03125 disk.drbd2165.in‚0$ memory_percent‚93.81320476531982 disk.sda3.out‚ 102.70703125 disk.dm-1.out‚ 102.70703125 disk.dm-0.in‚ 3771.5859375 network.recv‚770251 disk.sda.out‚ 102.70703125( disk.nvme0c0n1.out‚22336.44384765625& disk.nvme0n1.out‚22336.44384765625" proc.memory.rssMB‚ 2847.7421875* disk./.usagePercent‚39.115961797367866 disk.nvme0n1.in‚ 3771.5859375 disk.sda1.out‚0 disk.sda2.in‚0Ï„/T”ó ‚(:‰ ¶„ëÒ“„©Ç gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚275.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@Ïßô ‚(:Ô ¶„ëÒÁˆ©Ë network.recv‚773476$ disk./.usageGB‚582.8573570251465 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 3771.5859375) proc.memory.availableMB‚ 1013.39453125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 3771.5859375% disk.nvme0n1.out‚22342.5146484375" proc.memory.rssMB‚ 2847.7421875" disk.nvme0c0n1.in‚ 3771.5859375 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 105.37109375 disk.sda.out‚ 105.37109375" disk.dm-0.out‚22342.5146484375 proc.cpu.threads‚17$ memory_percent‚93.81473064422607 network.sent‚ 3.201912e+06 disk.sda3.out‚ 105.37109375 disk.rbd1.out‚8.09375 disk.sda1.in‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚22342.5146484375 disk.sda.in‚ 0.47265625) proc.memory.percent‚17.38123893737793 cpu‚25.46011435623346) disk./.usagePercent‚39.11604627985417 disk.sda3.in‚ 0.47265625ÿEè.’õ ‚(:‡ Å„ëÒÛôÀÈ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚267.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T‹)×ö ‚(:Ì Å„ëÒ䦿Ë$ memory_percent‚93.81322860717773 disk.sda2.out‚0 disk.sda2.in‚0) proc.memory.percent‚17.38123893737793 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0n1.in‚ 3772.41796875" proc.memory.rssMB‚ 2847.7421875 network.sent‚ 3.208863e+06 disk.sda3.in‚ 0.47265625 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 3772.41796875 disk.dm-1.in‚ 0.47265625) disk./.usagePercent‚39.16977534908778 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625 disk.sda.out‚106.0625 cpu‚25.459652584468646 disk.dm-0.in‚ 3772.41796875 disk.dm-1.out‚106.0625 network.recv‚776923 disk.sda1.in‚0 disk.sda3.out‚106.0625 disk.drbd2165.in‚0# disk.dm-0.out‚22709.54736328125& disk.nvme0n1.out‚22709.54736328125' proc.memory.availableMB‚ 1013.640625# disk./.usageGB‚583.657958984375 disk.rbd1.out‚8.09375( disk.nvme0c0n1.out‚22709.54736328125 proc.cpu.threads‚17—e†½”÷ ‚(:‰ Ô„ëÒ‡ ºÉ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚263.118' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.6236 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ<)Þø ‚(:Ó Ô„ëÒîêÆË network.recv‚780322 disk.sda3.out‚ 107.04296875 disk.rbd0.out‚0 disk.rbd1.in‚0 network.sent‚ 3.213815e+06" proc.memory.rssMB‚ 2847.7421875( proc.memory.availableMB‚ 1013.8828125$ disk./.usageGB‚583.6289329528809$ disk.nvme0n1.out‚23186.916015625 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 disk.dm-1.out‚ 107.04296875 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.in‚0 cpu‚25.45919721033063! disk.dm-0.out‚23186.916015625 disk.sda.in‚ 0.47265625# disk.nvme0c0n1.in‚ 3772.41796875 proc.cpu.threads‚17# memory_percent‚93.8117504119873 disk.sda1.in‚0 disk.sda.out‚ 107.04296875& disk.nvme0c0n1.out‚23186.916015625) proc.memory.percent‚17.38123893737793) disk./.usagePercent‚39.16782738775974! disk.nvme0n1.in‚ 3772.41796875 disk.sda1.out‚0 disk.dm-1.in‚ 0.47265625 disk.rbd1.out‚8.171875 disk.drbd2165.out‚0 disk.dm-0.in‚ 3772.41796875B:[2ù - _wandbruntime‚3058  _runtime‚3058ùÞ@4Çú Á & lr-AdamW/pg1‚0.0004768560374267314 & lr-AdamW/pg2‚0.0004768560374267314 trainer/global_step‚1749 % _timestamp‚1.7843328983619416e+09 _runtime‚3058.947012756 _step‚1402¢fÌû Æ _runtime‚3058.947012756 _step‚140 & lr-AdamW/pg1‚0.0004768560374267314 & lr-AdamW/pg2‚0.0004768560374267314 trainer/global_step‚1749 % _timestamp‚1.7843328983619416e+09ŒÞ£±„2ü - _wandbruntime‚3058  _runtime‚3058UaÛ%í'ý çO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚503545774210560 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15133538 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4490126 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1643723143680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138108915385344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚503325516234720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1643723143680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚501683261370368 _step‚141 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1645068533248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1642254864352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚501683235942400 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚501900679503872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1645068533248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14892889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135985413787648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚3058.948310746 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14892889 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚503324146206430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15133538 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚501900705677312 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚501900705677312 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4963310 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15143511 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30036406 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4963044 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4490121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843328983630774e+09 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚138109318012928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚503544402647552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2123501597696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚501900679503872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135985781878784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚503544402647552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1640910264030¾WùlDþ ìO [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1749 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚501900679503872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135985781878784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚503544402647552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1640910264030 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚503545774210560 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15133538 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4490126 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1643723143680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138108915385344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚503325516234720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1643723143680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚501683261370368 _step‚141 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1645068533248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1642254864352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚501683235942400 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚501900679503872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1645068533248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14892889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135985413787648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _ruH-—”®ntime‚3058.948310746 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14892889 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚503324146206430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15133538 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚501900705677312 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚501900705677312 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4963310 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15143511 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30036406 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4963044 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4490121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843328983630774e+09 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚138109318012928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚503544402647552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2123501597696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚502178680233984 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14901101 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚136062900195328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4966086 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚502178706407424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15152003 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2124689546240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9458492 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚502178706407424 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚1749 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚503824699983360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚503823328413184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15142016 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4492669 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚503602951450852 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚142 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30043111 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚136063268286464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15142016 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9458755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30053104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15152003 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30043111 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14901095 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14901095 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2124724075520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4492664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1645993575936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4965828 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138187992361984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1643178330710 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1641833730276 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 % _timestamp‚1.7843328997828214e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1644648179200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30053104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416ß,Õ”ü&ƒ öM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚503824699983360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚503823328413184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15142016 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4492669 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚503602951450852 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚142 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30043111 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚136063268286464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15142016 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9458755 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30053104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15152003 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30043111 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14901095 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14901095 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2124724075520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4492664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1645993575936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4965828 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138187992361984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1643178330710 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1641833730276 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 % _timestamp‚1.7843328997828214e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1644648179200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30053104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚3060.368204083 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚502178680233984 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14901101 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚136062900195328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4966086 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚502178706407424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15152003 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2124689546240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9458492 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚502178706407424 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚1749 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627ŽM´ ¦2„ - _wandbruntime‚3060  _runtime‚3060™KDê… € % _timestamp‚1.7843328998504033e+09 _step‚143 0loss/FRt1/assignment_loss‚2.0852816337521807 2loss/SRqqt2/assignment_loss‚0.6758717188733804 1loss/FBt1/assignment_loss‚0.05896618192424237 .loss/FBt1/detection_loss‚0.564589998458882 /loss/FBt2/detection_loss‚1.3396649139268058 trainer/global_step‚1749 0loss/FRt2/assignment_loss‚0.9299540832115892 2loss/FBt2/assignment_loss‚0.021442121340041555 /loss/FRt1/detection_loss‚0.5837570736909811 2loss/SRqqt1/assignment_loss‚0.6831263138488909 &loss/total_loss‚10.423238532986684 epoch‚4 _runtime‚3060.435592755 /loss/FRt2/detection_loss‚1.6310538194497655 0loss/SRqqt1/detection_loss‚1.130680001500222 0loss/SRqqt2/detection_loss‚0.718850673009703pQ- † … 2loss/SRqqt2/assignment_loss‚0.6758717188733804 1loss/FBt1/assignment_loss‚0.05896618192424237 .loss/FBt1/detection_loss‚0.564589998458882 /loss/FBt2/detection_loss‚1.3396649139268058 trainer/global_step‚1749 0loss/FRt2/assignment_loss‚0.9299540832115892 2loss/FBt2/assignment_loss‚0.021442121340041555 /loss/FRt1/detection_loss‚0.5837570736909811 2loss/SRqqt1/assignment_loss‚0.6831263138488909 &loss/total_loss‚10.423238532986684 epoch‚4 _runtime‚3060.435592755 /loss/FRt2/detection_loss‚1.6310538194497655 0loss/SRqqt1/detection_loss‚1.130680001500222 0loss/SRqqt2/detection_loss‚0.718850673009703 % _timestamp‚1.7843328998504033e+09 _step‚143 0loss/FRt1/assignment_loss‚2.0852816337521807…°KÓŸ‡ ‚(:” ò„ëÒúãÇ gpu.0.gpu‚82.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.182' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.836400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀlúùÞˆ ‚(:Ó ò„ëÒ¶çºÉ disk.drbd2165.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 3838.96875) proc.memory.availableMB‚ 1013.04296875) disk./.usagePercent‚39.11999340401708$ disk./.usageGB‚582.9161720275879 disk.rbd1.out‚8.234375 disk.sda.in‚ 0.47265625 disk.nvme0c0n1.in‚ 3838.96875 network.recv‚788108 disk.sda3.out‚ 108.26953125 disk.dm-1.out‚ 108.26953125 disk.sda1.out‚0# proc.memory.rssMB‚ 2847.74609375 disk.sda3.in‚ 0.47265625 disk.rbd1.in‚0 disk.sda.out‚ 108.26953125# disk.dm-0.out‚23206.33837890625 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 3.26911e+06 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚23206.33837890625 disk.dm-0.in‚ 3838.96875 cpu‚25.458337652288993 disk.dm-1.in‚ 0.47265625& disk.nvme0n1.out‚23206.33837890625$ memory_percent‚93.81687641143799) proc.memory.percent‚17.38126277923584 disk.rbd0.out‚0a>ì­ž‰ ‚(:“ …ëÒ‰¾ÓÈ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚279.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.85099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ìß~ߊ ‚(:Ô …ëÒôÿïÉ disk.sda3.out‚ 109.37890625( disk.nvme0c0n1.out‚23223.41748046875# proc.memory.rssMB‚ 2847.74609375$ memory_percent‚93.81678104400635 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 109.37890625 disk.nvme0c0n1.in‚ 3838.96875 disk.dm-0.in‚ 3838.96875) disk./.usagePercent‚39.11853672114726& disk.nvme0n1.out‚23223.41748046875 disk.sda3.in‚ 0.47265625) proc.memory.percent‚17.38126277923584 network.recv‚792117 disk.sda1.out‚0 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0 cpu‚25.457889140478414 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 1013.05859375 network.sent‚ 3.321758e+06$ disk./.usageGB‚582.8944664001465 disk.dm-1.in‚ 0.47265625 disk.rbd1.out‚8.234375 disk.sda.out‚ 109.37890625 disk.nvme0n1.in‚ 3838.96875 disk.sda1.in‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17# disk.dm-0.out‚23223.41748046875¹8PŸ‹ ‚(:” …ëÒÌá¿Ç gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚292.448' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.489599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡G²hÒŒ ‚(:Ç …ëÒüÐâÉ disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 3839.93359375) proc.memory.percent‚17.38126277923584 proc.cpu.threads‚17 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2847.74609375 network.recv‚794956 disk.drbd2165.in‚0% disk.nvme0c0n1.out‚23231.91015625# memory_percent‚93.8157320022583! disk.nvme0n1.in‚ 3839.93359375 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 111.09375 disk.dm-1.in‚ 0.47265625 cpu‚25.45753373991295 network.sent‚ 3.325846e+06$ disk./.usageGB‚582.0150337219238 disk.dm-1.out‚ 111.09375 disk.sda.out‚ 111.09375 disk.dm-0.out‚23231.91015625 disk.sda1.out‚0 disk.rbd1.out‚8.3125 disk.sda.in‚ 0.47265625) disk./.usagePercent‚39.05951725621857 disk.sda1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 3839.93359375) proc.memory.availableMB‚ 1013.23046875# disk.nvme0n1.out‚23231.91015625 disk.rbd0.in‚0ûËcÀŸ ‚(:” Ÿ…ëÒÆÌÇ gpu.0.gpu‚86.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚279.015' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.803000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lq*}ùŽ ‚(:× Ÿ…ëÒººæÉ disk.dm-1.in‚ 0.47265625 disk.drbd2165.in‚0) proc.memory.percent‚17.37678050994873 disk.dm-1.out‚ 111.99609375 disk.rbd1.in‚0& disk.nvme0n1.out‚23255.39306640625 disk.sda3.out‚ 111.99609375 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.in‚ 0.47265625 disk.nvme0n1.in‚ 3884.3671875 disk.rbd0.out‚0 cpu‚25.457099792718417 network.sent‚ 3.330819e+06$ disk./.usageGB‚582.0154571533203 disk.drbd2165.out‚0" disk.n>ÕjTévme0c0n1.in‚ 3884.3671875( disk.nvme0c0n1.out‚23255.39306640625 disk.sda1.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 111.99609375# disk.dm-0.out‚23255.39306640625 disk.sda3.in‚ 0.47265625$ memory_percent‚93.81518363952637( proc.memory.availableMB‚ 1013.3203125 network.recv‚798376 disk.sda2.out‚0 disk.rbd1.out‚8.3125 disk.dm-0.in‚ 3884.3671875# proc.memory.rssMB‚ 2847.01171875) disk./.usagePercent‚39.05954567305487>]ée” ‚(:‰ ®…ëÒãÙºÇ gpu.0.gpu‚87.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚283.649' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.7298 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚ÓHØ ‚(:Í ®…ëÒáѳÉ$ disk./.usageGB‚582.0152397155762 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 112.31640625 disk.nvme0n1.in‚ 3884.3671875 disk.drbd2165.out‚0$ memory_percent‚93.81494522094727# proc.memory.rssMB‚ 2847.02734375 network.sent‚ 3.337977e+06 network.recv‚801709 disk.dm-0.out‚23258.44921875 disk.sda3.out‚ 112.31640625 disk.rbd1.in‚0 disk.dm-0.in‚ 3884.3671875# disk.nvme0n1.out‚23258.44921875 disk.sda1.in‚0 disk.sda.out‚ 112.31640625 cpu‚25.456598097099178 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 3884.3671875 proc.cpu.threads‚17 disk.sda3.in‚ 0.47265625 disk.dm-1.in‚ 0.47265625% disk.nvme0c0n1.out‚23258.44921875) disk./.usagePercent‚39.05953108062541 disk.sda1.out‚0 disk.rbd1.out‚8.4375' proc.memory.availableMB‚ 1013.359375) proc.memory.percent‚17.37687587738037‹Å2‘ - _wandbruntime‚3142  _runtime‚3142´T\É’ à ' lr-AdamW/pg1‚0.00047643368970455606 ' lr-AdamW/pg2‚0.00047643368970455606 trainer/global_step‚1799 % _timestamp‚1.7843329820452158e+09 _runtime‚3142.630295799 _step‚144P\y¢Î“ È ' lr-AdamW/pg1‚0.00047643368970455606 ' lr-AdamW/pg2‚0.00047643368970455606 trainer/global_step‚1799 % _timestamp‚1.7843329820452158e+09 _runtime‚3142.630295799 _step‚144˜nÚ2” - _wandbruntime‚3142  _runtime‚3142Waï½í'• çO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5101830 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚517264472228338 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1691384463872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚515577350279168 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚515577324851200 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843329820465055e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1690039074304 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9719381 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2182913056768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15559290 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1691384463872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚142043015319552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30872458 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139859734171648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚515800742182912 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4617276 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1688491977460 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚517492126646784 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9719101 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5102105 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚517265842256628 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚515800742182912 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚515800716009472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15303189 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚517490755083776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1799 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚517490755083776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1799 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚515800742182912 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4617276 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1688491977460 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚517492126646784 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9719101 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5102105 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚517265842256628 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚515800742182912 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚515800716009472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15303189 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0‘ ]·ý2— - _wandbruntime‚3144  _runtime‚3144ôˆ²çR&˜ ñM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚517544647501162 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1692309506560 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15577755 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30889156 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843329834465551e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9724312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139937220579328 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15311401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15311395 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9724585 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1689415443818 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30879163 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _step‚146 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2184051794944 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3144.031935232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚517771052419584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30879163 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15577755 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚516078742913024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1690964109824 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5104498 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15567768 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15311401 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15311395 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚517769680849408 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4619819 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 tæyŽ¥rainer/global_step‚1799 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0=5ƒ¿ü&™ öM M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚517544647501162 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1692309506560 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15577755 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30889156 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843329834465551e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9724312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139937220579328 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15311401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15311395 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9724585 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1689415443818 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30879163 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _step‚146 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2184051794944 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3144.031935232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚517771052419584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30879163 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15577755 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚516078742913024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1690964109824 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5104498 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15567768 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15311401 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15311395 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚517769680849408 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4619819 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 trainer/global_step‚1799’Ž©ûƒ2š - _wandbruntime‚3144  _runtime‚3144ÝÜx› ÿ 0loss/FRt2/assignment_loss‚1.0135548387964566 .loss/FRt1/detection_loss‚0.602265096736052 1loss/SRqqt2/detection_loss‚0.7245402779370066 _runtime‚3144.112393738 /loss/FRt1/assignment_loss‚2.144040020816452 1loss/FBt1/assignment_loss‚0.03534708262010667 /loss/FBt2/detection_loss‚0.7693570790921941 &loss/total_loss‚10.197619493492418 _step‚147 2loss/SRqqt1/assignment_loss‚0.7060342347654763 2loss/SRqqt2/assignment_loss‚0.6917796432507091 /loss/FRt2/detection_loss‚1.5642043442698195 1loss/SRqqt1/detection_loss‚1.3474887561917837 epoch‚4 trainer/global_step‚1799 $ _timestamp‚1.784332983527364e+09 1loss/FBt2/assignment_loss‚0.09499920665988985 /loss/FBt1/detection_loss‚0.5040089123564717 Í­ œ „ 1loss/FBt2/assignment_loss‚0.09499920665988985 /loss/FBt1/detection_loss‚0.5040089123564717 0loss/FRt2/assignment_loss‚1.0135548387964566 .loss/FRt1/detection_loss‚0.602265096736052 1loss/SRqqt2/detection_loss‚0.7245402779370066 _runtime‚3144.112393738 /loss/FRt1/assignment_loss‚2.144040020816452 1loss/FBt1/assignment_loss‚0.03534708262010667 /loss/FBt2/detection_loss‚0.7693570790921941 &loss/total_loss‚10.197619493492418 _step‚147 2loss/SRqqt1/assignment_loss‚0.7060342347654763 2loss/SRqqt2/assignment_loss‚0.6917796432507091 /loss/FRt2/detection_loss‚1.5642043442698195 1loss/SRqqt1/detection_loss‚1.3474887561917837 epoch‚4 trainer/global_step‚1799 $ _timestamp‚1.784332983527364e+09“ð¼pï’ ‚(:‡ ½…ëÒÌÈ gpu.0.gpu‚86.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚344.93' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.986 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˃û¦Õž ‚(:Ê ½…ëÒѧÞÉ% proc.memory.availableMB‚ 1013.1875 disk.rbd1.in‚0 disk.sda.out‚ 112.64453125# disk.nvme0c0n1.in‚ 3885.33203125) proc.memory.percent‚17.37687587738037 disk.sda3.out‚ 112.64453125 disk.sda2.in‚0# proc.memory.rssMB‚ 2847.02734375 disk.dm-1.out‚ 112.64453125 disk.drbd2165.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.05941971734802 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚8.4375" disk.nvme0n1.out‚ 23293.4140625$ memory_percent‚93.81599426269531 cpu‚25.45624659812751 disk.sda1.out‚0 disk.dm-1.in‚ 0.47265625 disk.drbd2165.in‚0 disk.dm-0.in‚ 3885.33203125 disk.dm-0.out‚ 23293.4140625! disk.nvme0n1.in‚ 3885.33203125 network.sent‚ 3.340899e+06 network.recv‚805231 disk.sda.in‚ 0.47265625$ disk.nvme0c0n1.out‚ 23293.4140625$ disk./.usageGB‚582.0135803222656 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 disk.sda2.out‚0§- ”Ÿ ‚(:‰ Ì…ëÒ®Á­Ç gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚337.146' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.4292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š¹‰æ  ‚(:Û Ì…ëÒ³–®É$ memory_percent‚93.82061958312988) proc.memory.percent‚17.38126277923584 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 3885.33203125! disk.nvme0n1.in‚ 3885.33203125( proc.memory.availableMB‚ 1012.4296875# proc.memory.rssMB‚ 2847.74609375 cpu‚25.455831473588983 disk.sda3.in‚ 0.47265625 disk.rbd0.out‚0 disk.rbd1.out‚8.515625 disk.sda.out‚ 112.82421875 disk.drbd2165.out‚0 disk.dm-0.in‚ 3885.33203125# disk.dm-0.out‚23296.96630859375 network.recv‚809536) disk./.usagePercent‚39.05942355746104( disk.nvme0c0n1.out‚23296.96630859375 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 112.82421875 disk.rbd0.in‚0 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 112.82421875 disk.rbd1.in‚0& disk.nvme0n1.out‚23296.96630859375$ disk./.usageGB‚582.0136375427246 disk.sda.in‚ 0.47265625 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 3.40394e+06 disk.sda2.out‚0æK,ž¡ ‚(:“ Û…ëÒçòÔÇ gpu.0.gpu‚80.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚328.786' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.75720000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«L`=ᢠ‚(:Ö Û…ëÒ˜†ÛÉ disk.sda3.out‚ 113.1015625 disk.dm-1.out‚ 113.1015625 disk.sda.in‚ 0.47265625! proc.memory.rssMB‚ 2846.828125# disk.dm-0.out‚23306.51708984375& disk.nvme0n1.out‚23306.51708984375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 disk.sda.out‚ 113.1015625 cpu‚25.455486376008242* disk./.usagePercent‚39.059430469664456# disk.nvme0c0n1.in‚ 3920.74609375 disk.rbd1.out‚8.515625# memory_percent‚93.8161849975586* proc.memory.percent‚17.375659942626953( disk.nvme0c0n1.out‚23306.51708984375 disk.sda2.out‚0 disk.drbd2165.in‚0 network.recv‚812815$ disk./.usageGB‚582.0137405395508 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 3920.74609375 network.sent‚ 3.408891e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.rbd0.out‚0 disk.rbd1.in‚0& proc.memory.availableMB‚ 1013.15625 proc.cpu.threads‚17 disk.dm-0.in‚ 3920.74609375GûO³œ£ ‚(:‘ ê…ëÒÌ€·Ç gpu.0.gpu‚8.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚329.609' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.92179999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00­§ç㤠‚(:Ø ê…ëÒ·öºÉ* disk./.usagePercent‚39.059691597349385 disk.dm-0.in‚ 3921.7109375 disk.sda1.in‚0 disk.sda.out‚ 113.3046875( disk.nvme0c0n1.out‚23314.62841796875) proc.memory.availableMB‚ 1012.89453125 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625) proc.memory.percent‚17.37687587738037 proc.cpu.threads‚17 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 113.3046875 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚8.578125$ disk./.usageGB‚582.0176315307617 disk.drbd2165.out‚0$ memory_percent‚93.81778240203857 network.recv‚816226 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 3921.7109375# proc.memory.rssMB‚ 2847.02734375 network.sent‚ 3.413903e+06# disk.dm-0.out‚23314.62841796875 disk.dm-1.out‚ 113.3046875 disk.nvme0n1.in‚ 3921.7109375& disk.nvme0n1.out‚23314.62841796875 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 cpu‚25.455076812273777œ²jŸ¥ ‚(:” ù…ëÒàŒíÈ gpu.0.gpu‚69.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02Jµâ¦ ‚(:× ù…ëÒòß´É disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 113.64453125 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚582.0174598693848 disk.rbd0.in‚0# disk.dm-0.out‚23321.54833984375 disk.sda3.in‚ 0.47265625 network.recv‚820090 disk.dm-0.in‚ 3921.7109375 disk.nvme0n1.in‚ 3921.7109375 disk.rbd1.out‚8.578125# proc.memory.rssMB‚ 2847.02734375 network.sent‚ 3.41909e+06) proc.memory.availableMB‚ 1012.87890625) disk./.usagePercent‚39.05968007701034 disk.sda.out‚ 113.64453125) proc.memory.percent‚17.37687587738037 cpu‚25.4544359492206" disk.nvme0c0n1.in‚ 3921.7109375( disk.nvme0c0n1.out‚23321.54833984375 disk.rbd0.out‚0 disk.sda.in‚ 0.47265625& disk.nvme0n1.out‚23321.54833984375 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚93.81787776947021 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.out‚ 113.64453125 disk.sda2.out‚0u|ÒŸ§ ‚(:” ˆ†ëÒ÷¢µÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…šU)ç¨ ‚(:Ü ˆ†ëÒîÖìÉ cpu‚25.454100579599068( disk.nvme0c0n1.out‚23330.26806640625$ disk./.usageGB‚582.0176773071289 disk.sda3.in‚ 0.47265625 disk.rbd0.out‚0) proc.memory.availableMB‚ 1013.83984375 network.sent‚ 3.423996e+06 disk.rbd1.out‚8.65625! disk.nvme0n1.in‚ 3922.54296875 disk.sda2.in‚0) proc.memory.percent‚17.38126277923584) disk./.usagePercent‚39.05969466943979# disk.dm-0.out‚23330.26806640625 disk.sda1.in‚0 disk.rbd0.in‚0 network.recv‚823311 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 3922.54296875 disk.sda2.out‚0# proc.memory.rssMB‚ 2847.74609375& disk.nvme0n1.out‚23330.26806640625$ memory_percent‚93.81201267242432 disk.dm-0.in‚ 3922.54296875 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 113.98046875 proc.cpu.threads‚17 disk.sda.out‚ 113.98046875 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 113.98046875ÅJ2© - _wandbruntime‚3226  _runtime‚3226úÜ<ƪ À trainer/global_step‚1849 % _timestamp‚1.7843330662461324e+09 _runtime‚ 3226.83133392 _step‚148 & lr-AdamW/pg1‚0.0004759962357191131 & lr-AdamW/pg2‚0.0004759962357191131£9³SË« Å & lr-AdamW/pg2‚0.0004759962357191131 trainer/global_step‚1849 % _timestamp‚1.7843330662461324e+09 _runtime‚ 3226.83133392 _step‚148 & lr-AdamW/pg1‚0.0004759962357191131”YƒáÛ2¬ - _wandbruntime‚3226  _runtime‚3226ã,ØQì'­ æO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15995015 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9984355 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚531204798250246 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1737700394496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15713489 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1733384490246 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31708510 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15985042 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚ 3226.83305588 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31698531 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9984646 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚145976035879424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚529471413760000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚149 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15713495 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5240220 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚531437107520000 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚529471439187968 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚531438479083008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1737700394496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145976440604160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚143734054555648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1734729090568 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚529700752515072 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1736355004928 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4744426 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚531437107520000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143734422646784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚529700778688512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15713495 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290nž7E¶® ëO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31708510 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15985042 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚ 3226.83305588 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31698531 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9984646 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚145976035879424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚529471413760000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚149 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15713495 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5240220 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.oêßq";!n_train_batch_start/segment.all.peak‚1995 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚531437107520000 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚529471439187968 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚531438479083008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1737700394496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145976440604160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚143734054555648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1734729090568 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚529700752515072 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1736355004928 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4744426 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚531437107520000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143734422646784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚529700778688512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15713495 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15995015 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9984355 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚531204798250246 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1737700394496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15713489 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1733384490246 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31715215 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16003507 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚531717404855808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1737280040448 _step‚150 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146054706188800 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1734307956492 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15721701 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚529978779418624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16003507 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚529749320966144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚529978753245184 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143811540963328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15721695 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚531716033285632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4746964 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1737280040448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9989927 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15721695 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1849 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31725208 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9989649 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚531717404855808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146055110906368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚531483603494668 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640Öd&˜û&± õM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1849 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31725208 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9989649 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚531717404855808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146055110906368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚531483603494668 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2243165225472 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚531484973523070 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31715215 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16003507 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚531717404855808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1737280040448 _step‚150 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146054706188800 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1734307956492 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15721701 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚529978779418624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16003507 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚529749320966144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚529978753245184 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143811540963328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15721695 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚531716033285632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4746964 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1737280040448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9989927 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15721695 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚545384831519232 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚545146494300444 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚543600815194112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16410794 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5378381 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚3310.996686022 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16123795 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4871571 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149909410109952 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚543600789020672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10249952 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1784016325120 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16123795 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 $ _timestamp‚1.784333150411187e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1779621603354 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16410794 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 trainer/global_step‚1899 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1782670935552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1780966203676 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16123789 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545383459956224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1782670935552 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚545384831519232 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚545146494300444 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚543600815194112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16410794 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5378381 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚3310.996686022 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16123795 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4871571 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149909410109952 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚543600789020672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10249952 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1784016325120 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16123795 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 $ _timestamp‚1.784333150411187e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1779621603354 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16410794 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 trainer/global_step‚1899 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1782670935552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1780966203676 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16123789 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545383459956224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1782670935552 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10255272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16429259 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16419272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16132001 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚545425299544978 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚543643384446976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16132001 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _step‚154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1784941367808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚543643409874944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10255536 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16429259 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚543878815924224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1783595971072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32561260 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4874114 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1780545069600 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚543878815924224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149988488372736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allo]ﶄ cated‚ 1781889670034 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚545663757292032 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 trainer/global_step‚1899 % _timestamp‚1.7843331518566005e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚543878789750784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149988085752320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚545663757292032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1784941367808 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5381417 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147686229438464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚545423929516576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16131995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚543878789750784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 17835959710726®À#þ&Ç øM M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚545663757292032 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 trainer/global_step‚1899 % _timestamp‚1.7843331518566005e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚543878789750784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149988085752320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚545663757292032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1784941367808 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5381417 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147686229438464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚545423929516576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16131995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚543878789750784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1783595971072 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16131995 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10255272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16429259 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16419272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16132001 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚545425299544978 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚543643384446976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16132001 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _step‚154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1784941367808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚543643409874944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10255536 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16429259 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚543878815924224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1783595971072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32561260 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4874114 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1780545069600 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚543878815924224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149988488372736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1781889670034 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648šÍ)a2È - _wandbruntime‚3312  _runtime‚3312ñV]uÉ ý .loss/FBt1/assignment_loss‚0.02774073828123 1loss/FBt2/assignment_loss‚0.13841996788685265 1loss/SRqqt1/detection_loss‚1.1067353600488048 1loss/SRqqt2/detection_loss‚0.7010115787229161 /loss/FBt1/detection_loss‚0.5109168710580835 /loss/FBt2/detection_loss‚1.3718183358199894 %loss/total_loss‚10.49958939105234 trainer/global_step‚1899 .loss/FRt2/detection_loss‚1.648147256393775 epoch‚4 _runtime‚3312.499230595 _step‚155 0loss/FRt1/assignment_loss‚2.1436132679019515 0loss/FRt2/assignment_loss‚1.0063051676093522 2loss/SRqqt2/assignment_loss‚0.6075371878328115 % _timestamp‚1.7843331519140177e+09 2loss/SRqqt1/assignment_loss‚0.6570516683623511 /loss/FRt1/detection_loss‚0.5802919911342231dŒ’…Ê ‚ .loss/FBt1/assignment_loss‚0.02774073828123 1loss/FBt2/assignment_loss‚0.13841996788685265 1loss/SRqqt1/detection_loss‚1.1067353600488048 1loss/SRqqt2/detection_loss‚0.7010115787229161 /loss/FBt1/detection_loss‚0.5109168710580835 /loss/FBt2/detection_loss‚1.3718183358199894 %loss/total_loss‚10.49958939105234 trainer/global_step‚1899 .loss/FRt2/detection_loss‚1.648147256393775 epoch‚4 _runtime‚3312.499230595 _step‚155 0loss/FRt1/assignment_loss‚2.1436132679019515 0loss/FRt2/assignment_loss‚1.0063051676093522 2loss/SRqqt2/assignment_loss‚0.6075371878328115 % _timestamp‚1.7843331519140177e+09 2loss/SRqqt1/assignment_loss‚0.6570516683623511 /loss/FRt1/detection_loss‚0.5802919911342231›ÈgD<ŸË ‚(:” â†ëÒ»˜˜È gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚247.509' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.501799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j¦u&ÜÌ ‚(:Ñ â†ëÒÖÇØÉ disk.sda3.out‚ 117.26171875 network.sent‚ 3.512317e+06$ disk.nvme0n1.out‚23379.775390625# disk.nvme0c0n1.in‚ 4010.34765625) proc.memory.percent‚17.38128662109375! disk.dm-0.out‚23379.775390625 disk.dm-1.in‚ 0.47265625 disk.sda.in‚ 0.47265625 network.recv‚845012) disk./.usagePercent‚39.05969825354527! disk.nvme0n1.in‚ 4010.34765625 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0$ memory_percent‚93.81935596466064& disk.nvme0c0n1.out‚23379.775390625) proc.memory.availableMB‚ 1012.63671875 disk.rbd0.out‚0 disk.dm-0.in‚ 4010.34765625 disk.dm-1.out‚ 117.26171875 disk.drbd2165.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 disk.sda.out‚ 117.26171875 proc.memory.rssMB‚2847.75 cpu‚25.451747185484706 proc.cpu.threads‚17$ disk./.usageGB‚582.0177307128906 disk.rbd1.out‚8.859375 disk.drbd2165.out‚0 disk.sda2.out‚0'UKožÍ ‚(:“ ñ†ëÒЈÇÈ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚328.649' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.72980000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/VÛÎ ‚(:Ð ñ†ëÒ¬Ñ¬É disk.dm-1.in‚ 0.47265625' disk.nvme0c0n1.out‚23386.4306640625 disk.dm-0.in‚ 4010.34765625 network.recv‚849837! disk.nvme0n1.in‚ 4010.34765625" disk.dm-0.out‚23386.4306640625 disk.rbd1.out‚8.859375 disk.sda.in‚ 0.47265625# disk.nvme0c0n1.in‚ 4010.34765625) proc.memory.percent‚17.38128662109375 proc.cpu.threads‚17) disk./.usagePercent‚39.05971105392199 disk.sda3.out‚ 117.8203125 disk.sda2.out‚0 cpu‚25.45144682699496% disk.nvme0n1.out‚23386.4306640625$ memory_percent‚93.81964206695557 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2165.in‚0 network.sent‚ 3.575377e+06$ disk./.usageGB‚582.0179214477539 disk.sda1.in‚0 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1012.58984375 disk.dm-1.out‚ 117.8203125 disk.rbd1.in‚0 disk.sda.out‚ 117.8203125 proc.memory.rssMB‚2847.75 disk.sda3.in‚ 0.47265625 disk.rbd0.out‚0÷¶U¿”Ï ‚(:‰ €‡ëÒÔò Ç gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚334.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª:Ù·ÒÐ ‚(:Ç €‡ëÒÂñ¬Ê network.recv‚852737 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚23395.0732421875 disk.nvme0n1.in‚ 4011.3125) proc.memory.availableMB‚ 1012.41015625 disk.rbd0.out‚0 disk.rbd0.in‚0$ memory_percent‚93.82073879241943 proc.memory.rssMB‚2847.75) proc.memory.percent‚17.38128662109375$ disk./.usageGB‚582.0178871154785 disk.dm-1.in‚ 0.47265625 disk.rbd1.out‚8.9375 disk.sda.in‚ 0.47265625 disk.dm-0.in‚ 4011.3125 network.sent‚ 3.580354e+06 disk.sda2.in‚0 disk.dm-1.out‚ 118.23828125 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 proc.cpu.threads‚17 disk.sda.out‚ 118.23828125 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 4011.3125" disk.dm-0.out‚23395.0732421875 disk.sda2.out‚0% disk.nvme0n1.out‚23395.0732421875* disk./.usagePercent‚39.059708749854174 disk.sda3.out‚ 118.23828125 cpu‚25.451054037422477ãŠÊîžÑ ‚(:“ ‡ëÒ£Ç gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚332.025' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.40499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O•¤ÑÒ ‚(:Æ ‡ëÒ­ÿžË network.recv‚856087 disk.dm-1.in‚ 0.47265625 disk.sda.out‚ 119.390625# disk.dm-0.out‚23407.25634765625$ disk./.usageGB‚582.0177993774414 disk.sda3.out‚ 119.390625 disk.rbd0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 1012.390625 proc.memory.rssMB‚2847.75 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 119.390625 disk.rbd1.in‚0 disk.rbd1.out‚8.9375 disk.sda.in‚ 0.47265625 disk.dm-0.in‚ 4054.390625$ memory_percent‚93.82085800170898) disk./.usagePercent‚39.05970286168089 disk.sda3.in‚ 0.47265625 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 4054.390625( disk.nvme0c0n1.out‚23407.25634765625) proc.memory.percent‚17.38128662109375 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 3.58544e+06 disk.sda1.in‚0 cpu‚25.450740037592645 disk.nvme0n1.in‚ 4054.390625& disk.nvme0n1.out‚23407.25634765625 disk.sda2.out‚0Ò•²n”Ó ‚(:‰ ž‡ëÒн¡Ç gpu.0.gpu‚66.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚336.177' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.2354 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽÀ‰GÆÔ ‚(:» ž‡ëÒä…šË! disk.nvme0c0n1.in‚ 4054.390625 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 network.recv‚859528& disk.nvme0c0n1.out‚23411.060546875 disk.dm-0.in‚ 4054.390625 disk.sda3.in‚ 0.47265625 disk.dm-1.out‚ 120.03125 proc.cpu.threads‚17# disk./.usageGB‚582.017765045166 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0 disk.sda3.out‚ 120.03125$ memory_percent‚93.81628036499023 disk.sda.out‚ 120.03125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.out‚9 cpu‚25.45036934555076* disk./.usagePercent‚39.059700557613084 disk.nvme0n1.in‚ 4054.390625$ disk.nvme0n1.out‚23411.060546875 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625 proc.memory.rssMB‚ 2847.03125 disk.sda1.out‚0 network.sent‚ 3.590435e+06 disk.drbd2165.out‚0! disk.dm-0.out‚23411.060546875' proc.memory.availableMB‚ 1013.140625) proc.memory.percent‚17.37689971923828€y÷”Õ ‚(:‰ ­‡ëÒž€É gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚297.626' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.5252 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û•|3ÒÖ ‚(:Ç ­‡ëÒš±ÈÉ disk.sda3.in‚ 0.47265625 disk.dm-1.in‚ 0.47265625$ memory_percent‚93.81482601165771 proc.memory.rssMB‚ 2847.03125 disk.drbd2165.in‚0 disk.dm-1.out‚ 120.4921875) proc.memory.availableMB‚ 1013.37890625 disk.sda.in‚ 0.47265625# disk.nvme0c0n1.in‚ 4055.35546875% disk.nvme0c0n1.out‚23422.32421875# disk.nvme0n1.out‚23422.32421875 disk.sda2.in‚0 disk.rbd1.in‚0$ disk./.usageGB‚582.0178527832031 disk.sda3.out‚ 120.4921875 network.recv‚862705 disk.rbd0.out‚0! disk.nvme0n1.in‚ 4055.35546875 cpu‚25.449880770378233 network.sent‚ 3.595284e+06) disk./.usagePercent‚39.05970644578637 disk.dm-0.out‚23422.32421875 disk.sda1.in‚0 disk.sda2.out‚0) proc.memory.percent‚17.37689971923828 proc.cpu.threads‚17 disk.rbd1.out‚9 disk.sda.out‚ 120.4921875 disk.drbd2165.out‚0 disk.dm-0.in‚ 4055.35546875 disk.rbd0.in‚0 disk.sda1.out‚0ÀÐ2× - _wandbruntime‚3395  _runtime‚3395“Ø.ñÉØ Ã trainer/global_step‚1949 % _timestamp‚1.7843332346375134e+09 _runtime‚3395.222799666 _step‚156 ' lr-AdamW/pg1‚0.00047507612231427503 ' lr-AdamW/pg2‚0.00047507612231427503#@Ù ÎÙ È _step‚156 ' lr-AdamW/pg1‚0.00047507612231427503 ' lr-AdamW/pg2‚0.00047507612231427503 trainer/global_step‚1949 % _timestamp‚1.7843332346375134e+09 _runtime‚3395.222799666œ™4\®2Ú - _wandbruntime‚3395  _runtime‚3395€*´#%Û éO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33370635 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚153842827145728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚559086820322352 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16846519 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚559085450294062 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚151483063414784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚557259617005568 _runtime‚3395.224570901 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚559331183955456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚559331183955456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚157 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚557500851699712 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1827203316784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚557500851699712 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2360131822080 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚557500825526272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16836546 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1830332255744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10514777 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4998721 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚559329812392448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚559331183955456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚157 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚557500851699712 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1827203316784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚557500851699712 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2360131822080 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚557500825526272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16836546 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1830332255744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10514777 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4998721 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚559329812392448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10520040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1829911901696 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚557778852429824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33397312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16855011 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33387319 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚557537498783744 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153921894201856 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16542301 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2361307752960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1831257298432 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚1949 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16845024 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 % _timestamp‚1.7843332360598142e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚557778826256384 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1829911901696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10520307 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5519038 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16845024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5001269 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚158 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16855011 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5518776 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2361344379392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33397312 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5001264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚559364255538484 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚557778852429824 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16542301 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151560181731328 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1828126783142 _runtime‚3396.645240107 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153921489484288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151560549822464 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚559610109728256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚559608738158080 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚158 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16855011 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5518776 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2361344379392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33397312 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5001264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚559364255538484 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚557778852429824 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16542301 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151560181731328 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1828126783142 _runtime‚3396.645240107 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153921489484288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151560549822464 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚559610109728256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚559608738158080 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10520040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1829911901696 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚557778852429824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33397312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16855011 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33387319 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚557537498783744 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153921894201856 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16542301 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2361307752960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1831257298432 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚1949 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16845024 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 % _timestamp‚1.7843332360598142e+09žVkŽ2à - _wandbruntime‚3396  _runtime‚3396ÊaK"á  2loss/SRqqt1/assignment_loss‚0.6838987416320262 /loss/FRt1/detection_loss‚0.5686726936332468 /loss/FRt2/detection_loss‚1.5552320826414323 1loss/SRqqt1/detection_loss‚1.0949991729586048 /loss/FBt2/detection_loss‚1.2976413646413545 _step‚159 2loss/SRqqt2/assignment_loss‚0.6759216515238492 /loss/FBt1/detection_loss‚0.6358899913516257 epoch‚4 % _timestamp‚1.7843332361358638e+09 &loss/total_loss‚10.474141976331477 /loss/FRt1/assignment_loss‚2.068392920256847 0loss/FRt2/assignment_loss‚1.0273295179719018 1loss/FBt2/assignment_loss‚0.11068998898180585 _runtime‚3396.720907428 1loss/FBt1/assignment_loss‚0.08639326992844136 1loss/SRqqt2/detection_loss‚0.6690805808103405 trainer/global_step‚1949"­§÷ â † trainer/global_step‚1949 2loss/SRqqt1/assignment_loss‚0.6838987416320262 /loss/FRt1/detection_loss‚0.5686726936332468 /loss/FRt2/detection_loss‚1.5552320826414323 1loss/SRqqt1/detection_loss‚1.0949991729586048 /loss/FBt2/detection_loss‚1.2976413646413545 _step‚159 2loss/SRqqt2/assignment_loss‚0.6759216515238492 /loss/FBt1/detection_loss‚0.6358899913516257 epoch‚4 % _timestamp‚1.7843332361358638e+09 &loss/total_loss‚10.474141976331477 /loss/FRt1/assignment_loss‚2.068392920256847 0loss/FRt2/assignment_loss‚1.0273295179719018 1loss/FBt2/assignment_loss‚0.11068998898180585 _runtime‚3396.720907428 1loss/FBt1/assignment_loss‚0.08639326992844136 1loss/SRqqt2/detection_loss‚0.6690805808103405Ÿóí’æ“ã ‚(:‰ ¼‡ëÒØöôÇ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49;Yª616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚303.224' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.6448 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AUëÚä ‚(:Ï ¼‡ëÒÕŽÅÉ disk.sda2.in‚0 disk.dm-1.out‚ 121.1328125 cpu‚25.449590121194326 disk.drbd2165.out‚0& disk.nvme0n1.out‚23430.32080078125 disk.dm-1.in‚ 0.47265625 proc.memory.rssMB‚2847.75) proc.memory.percent‚17.38128662109375 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625$ memory_percent‚93.82143020629883 disk.drbd2165.in‚0 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚9.078125 disk.sda.out‚ 121.1328125 proc.cpu.threads‚17$ disk./.usageGB‚582.0172920227051" disk.nvme0c0n1.in‚ 4056.1953125( disk.nvme0c0n1.out‚23430.32080078125 disk.dm-0.in‚ 4056.1953125 disk.sda2.out‚0) disk./.usagePercent‚39.05966881267884# disk.dm-0.out‚23430.32080078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 121.1328125 network.sent‚ 3.600265e+06' proc.memory.availableMB‚ 1012.296875 network.recv‚865902 disk.nvme0n1.in‚ 4056.19531258hõ™Ÿå ‚(:” ˇëÒ¡˜éÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚256.408' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.281600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05W%Ìæ ‚(:Á ˇëÒ½£²É disk.dm-0.in‚ 4094.7578125$ disk.nvme0n1.out‚23436.134765625$ memory_percent‚93.81911754608154 disk.sda.in‚ 0.47265625 disk.sda3.in‚ 0.47265625 proc.memory.rssMB‚2847.75 disk.rbd0.in‚0 network.recv‚870618 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚23436.134765625 disk.sda3.out‚121.6875 network.sent‚ 3.663282e+06! disk.dm-0.out‚23436.134765625) proc.memory.percent‚17.38128662109375$ disk./.usageGB‚582.0173835754395 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 disk.rbd1.out‚9.078125 disk.sda.out‚121.6875 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0) proc.memory.availableMB‚ 1012.67578125 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 4094.7578125 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.44923142105721) disk./.usagePercent‚39.05967495685966 disk.dm-1.out‚121.6875 disk.nvme0n1.in‚ 4094.7578125 disk.sda2.out‚0RáҔç ‚(:‰ Ú‡ëÒÂеÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚282.783' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.5566 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´1¥Üè ‚(:Ñ Ú‡ëÒ®ÅÉ& disk.nvme0c0n1.out‚23444.208984375$ disk.nvme0n1.out‚23444.208984375) proc.memory.percent‚17.37689971923828 disk.drbd2165.in‚0 disk.sda3.out‚ 122.04296875 disk.sda2.out‚0 disk.dm-1.out‚ 122.04296875 disk.sda.in‚ 0.47265625 proc.cpu.threads‚17) disk./.usagePercent‚39.05967188476925" disk.nvme0c0n1.in‚ 4122.9296875 disk.rbd0.out‚0 disk.rbd1.out‚9.140625) proc.memory.availableMB‚ 1012.99609375 proc.memory.rssMB‚ 2847.03125$ disk./.usageGB‚582.0173377990723 disk.drbd2165.out‚0 disk.dm-0.in‚ 4122.9296875 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚93.81716251373291 cpu‚25.448871207926416 network.sent‚ 3.668384e+06 disk.nvme0n1.in‚ 4122.9296875 disk.sda.out‚ 122.04296875 network.recv‚873628 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 disk.dm-1.in‚ 0.47265625! disk.dm-0.out‚23444.208984375 disk.rbd1.in‚0ÉײŸé ‚(:” é‡ëÒ°´ÞÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚263.941' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.788199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£à èê ‚(:Ý é‡ëÒ½ÌÖÉ disk.rbd1.in‚0 disk.dm-1.in‚ 0.47265625 disk.sda.in‚ 0.47265625 disk.dm-1.out‚ 122.47265625 disk.sda3.in‚ 0.47265625$ memory_percent‚93.81744861602783 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 4123.76171875 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.percent‚17.37692356109619 disk.rbd1.out‚9.140625# disk.dm-0.out‚23453.66650390625 disk.sda1.in‚0 proc.cpu.threads‚17) proc.memory.availableMB‚ 1012.94921875$ disk./.usageGB‚582.0172424316406( disk.nvme0c0n1.out‚23453.66650390625 disk.dm-0.in‚ 4123.76171875& disk.nvme0n1.out‚23453.66650390625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2847.03515625 disk.sda.out‚ 122.47265625 cpu‚25.448514272214336) disk./.usagePercent‚39.05966548458089 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 4123.76171875 disk.sda1.out‚0 disk.sda3.out‚ 122.47265625 network.sent‚ 3.673345e+06 network.recv‚876788Uëc­”ë ‚(:‰ ø‡ëÒ§™±Ç gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚265.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž~c!âì ‚(:× ø‡ëÒÛ¾ÈÉ disk.dm-1.in‚ 0.47265625) disk./.usagePercent‚39.05966164446788 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 122.76953125$ memory_percent‚93.81804466247559 proc.cpu.threads‚17 network.recv‚880168$ disk./.usageGB‚582.0171852111816# disk.nvme0c0n1.in‚ 4123.76171875 disk.sda3.out‚ 122.76953125 disk.rbd1.out‚9.28125 cpu‚25.44816419303313 network.sent‚ 3.678405e+06" disk.dm-0.out‚23456.9638671875 disk.sda2.in‚0 disk.dm-1.out‚ 122.76953125! disk.nvme0n1.in‚ 4123.76171875 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 4123.76171875 disk.sda3.in‚ 0.47265625 disk.sda.in‚ 0.47265625' disk.nvme0c0n1.out‚23456.9638671875 disk.sda1.out‚0( proc.memory.availableMB‚ 1012.8515625% disk.nvme0n1.out‚23456.9638671875 disk.rbd0.out‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2847.75390625) proc.memory.percent‚17.38131046295166ŸïÝ12í - _wandbruntime‚3479  _runtime‚3479’0uÆî À & lr-AdamW/pg1‚0.0004745935219920445 & lr-AdamW/pg2‚0.0004745935219920445 trainer/global_step‚1999 $ _timestamp‚1.784333318495878e+09 _runtime‚3479.081056572 _step‚160“®DËï Å & lr-AdamW/pg1‚0.0004745935219920445 & lr-AdamW/pg2‚0.0004745935219920445 trainer/global_step‚1999 $ _timestamp‚1.784333318495878e+09 _runtime‚3479.081056572 _step‚160 `Ç.×2ð - _wandbruntime‚3479  _runtime‚3479ÙVʨï'ñ éO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚155357383798784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17272271 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10780202 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1872095829570 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚571400862031872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1876648186368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155357015707648 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1873440429892 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 trainer/global_step‚1999 _step‚161 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34216666 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16944395 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17272271 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5654592 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157776077241856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2419061534208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚573025776315970 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573276164828672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚571400862031872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5125876 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 % _timestamp‚1.7843333184973555e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚571153705914368 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573277536391680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1875302796800 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10780468 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚573027146344260 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17262298 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34206687 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚571400888205312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16944389 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16944395 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16944389 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573276164828672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1875302796800 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17262298 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚571153680486400 _runtime‚3479.082975725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5654331 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2419096070656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚571400888205312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157776479869440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973ÌêZô'ò îO MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1872095829570 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚571400862031872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1876648186368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155357015707648 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1873440429892 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 trainer/global_step‚1999 _step‚161 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34216666 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16944395 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17272271 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5654592 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157776077241856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2419061534208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚573025776315970 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573276164828672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚571400862031872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5125876 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 % _timestamp‚1.7843333184973555e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚571153705914368 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573277536391680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1875302796800 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10780468 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚573027146344260 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17262298 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34206687 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚571400888205312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16944389 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16944395 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16944389 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573276164828672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1875302796800 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17262298 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚571153680486400 _runtime‚3479.082975725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5654331 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2419096070656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚571400888205312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157776479869440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚155357383798784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17272271 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10780202 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _runtime‚3480.482104687 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16952595 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1874363896250 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17280763 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34233364 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚573556462164480 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7843333198965542e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚571678862761984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1873019295816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17270776 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34233364 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16952595 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1877573229056 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34223371 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34223371 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚571678888935424 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16952601 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1876227832320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚155434502115328 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚573304581560392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573555090594304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10785743 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17280763 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 trainer/global_step‚1999 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155434870206464 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5657072 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157855145739776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚571678888935424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚157854743119360 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 _step‚162 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1877573229056 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573556462164480 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5128419 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1876227832320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚571431562264576 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16952601 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10785486 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5657324 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚571678862761984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚573305951588794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚571431587692544+ýþ&÷ øM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10785486 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5657324 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚571678862761984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚573305951588794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚571431587692544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _runtime‚3480.482104687 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16952595 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1874363896250 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17280763 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34233364 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚573556462164480 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7843333198965542e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚571678862761984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1873019295816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17270776 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34233364 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16952595 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1877573229056 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34223371 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34223371 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚571678888935424 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16952601 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1876227832320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚155434502115328 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚573304581560392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573555090594304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10785743 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17280763 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 trainer/global_step‚1999 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155434870206464 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5657072 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157855145739776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚571678888935424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚157854743119360 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 _step‚162 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1877573229056 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573556462164480 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5128419 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1876227832320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚571431562264576 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16952601 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51¢Ïl$<2ø - _wandbruntime‚3480  _runtime‚3480—ö½ù  1loss/SRqqt2/detection_loss‚0.7233619115126756 %loss/total_loss‚10.58467229396356 0loss/FRt2/assignment_loss‚0.9490884028279218 /loss/FBt1/detection_loss‚0.5310742307159124 /loss/FBt2/detection_loss‚1.7162047477811575 _runtime‚3480.560019043 _step‚163 2loss/SRqqt2/assignment_loss‚0.6137786083763371 1loss/FBt1/assignment_loss‚0.03579857412642598 /loss/FRt1/detection_loss‚0.5974039270376056 /loss/FRt2/detection_loss‚1.6358157991983253 1loss/SRqqt1/detection_loss‚1.0530642828461332 % _timestamp‚1.7843333199749541e+09 /loss/FRt1/assignment_loss‚1.945949657633145 2loss/SRqqt1/assignment_loss‚0.7575789983627639 epoch‚4 trainer/global_step‚1999 2loss/FBt2/assignment_loss‚0.025553153545157272ô:¯ ú † 2loss/FBt2/assignment_loss‚0.025553153545157272 1loss/SRqqt2/detection_loss‚0.7233619115126756 %loss/total_loss‚10.58467229396356 0loss/FRt2/assignment_loss‚0.9490884028279218 /loss/FBt1/detection_loss‚0.5310742307159124 /loss/FBt2/detection_loss‚1.7162047477811575 _runtime‚3480.560019043 _step‚163 2loss/SRqqt2/assignment_loss‚0.6137786083763371 1loss/FBt1/assignment_loss‚0.03579857412642598 /loss/FRt1/detection_loss‚0.5974039270376056 /loss/FRt2/detection_loss‚1.6358157991983253 1loss/SRqqt1/detection_loss‚1.0530642828461332 % _timestamp‚1.7843333199749541e+09 /loss/FRt1/assignment_loss‚1.945949657633145 2loss/SRqqt1/assignment_loss‚0.7575789983627639 epoch‚4 trainer/global_step‚1999£&8À+žû ‚(:“ –ˆëÒ°Ç gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚328.649' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.72980000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚089ÏAÒü ‚(:Ç –ˆëÒͽåÈ disk.rbd0.out‚0 disk.dm-1.out‚ 123.6171875* proc.memory.percent‚17.397427558898926# disk.nvme0n1.out‚23471.76171875 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 123.6171875 disk.drbd2165.in‚0 disk.dm-0.in‚ 4177.640625 disk.sda2.out‚0 disk.dm-0.out‚23471.76171875 cpu‚25.457352948581352 proc.cpu.threads‚17) disk./.usagePercent‚39.05943200570967! disk.nvme0c0n1.in‚ 4177.640625 disk.drbd2165.out‚0# memory_percent‚93.8333511352539 network.sent‚ 3.734889e+06 disk.sda3.in‚ 0.47265625$ disk./.usageGB‚582.0137634277344% disk.nvme0c0n1.out‚23471.76171875 disk.dm-1.in‚ 0.47265625 disk.rbd1.out‚9.28125 disk.sda.in‚ 0.47265625# proc.memory.rssMB‚ 2850.39453125 disk.nvme0n1.in‚ 4177.640625 disk.sda3.out‚ 123.6171875 disk.rbd0.in‚0& proc.memory.availableMB‚ 1010.34375 network.recv‚887292 disk.sda1.out‚0Z&N‹2ý - _wandbruntime‚3508  _runtime‚3508¾'†3þ Æ{ 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.024294022470712662 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3878408670425415 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4480176568031311 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8063265681266785 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5285622477531433 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17329305410385132 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7152579426765442 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8448563814163208 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9744220972061157 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 *jet/accuracy_1_of_2‚0.6683337688446045 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9044681787490845 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07997293770313263 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.47544774413108826 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16918380558490753 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 jet/accuracy_3_of_5‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_4‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4982261061668396 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9979571104049683 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5499538779258728 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07456191629171371 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 )loss/val_total_loss‚9.800309887202689 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15437321364879608 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_1_of_1‚1 6validation_average_jet_accuracy‚0.2955716848373413 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9962121248245239 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5902647972106934 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2292509377002716 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15599854290485382 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0799809992313385 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.47017183899879456 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8063265681266785 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11230243742465973 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07456191629171371 particle/accuracy_2_of_2‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9965478777885437 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6097363233566284 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5762171149253845 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5505030155181885 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16987408697605133 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 /particle/accuracy_3_of_4‚0.9970238208770752 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7714150547981262 particle/accuracy_4_of_5‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4804166853427887 'particle/f_score‚0.6287062764167786 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9994933605194092 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_2_of_3‚0.4563818573951721 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6601851582527161 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 epoch‚4 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8458123803138733 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6057478785514832 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.47017183899879456 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.53125 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 +jet/accuracy_3_of_3‚0.06340016424655914 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.11449488997459412 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1208006888628006 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5799822807312012 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01777498796582222 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9936507940292358 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9946417212486267 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.47046512365341187 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11500000953674316 *jet/accuracy_1_of_3‚0.8874333500862122 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24586404860019684 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 (particle/accuracy‚0.8391191363334656 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9920565485954285 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5623694658279419 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9949482679367065 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9953996539115906 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6710695624351501 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_1_of_2‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9953996539115906 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_2_of_3‚1 /particle/accuracy_4_of_4‚0.9661495685577393 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3935185372829437 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8176871538162231 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6726835370063782 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7921391725540161 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1614629626274109 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5081256628036499 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.567078709602356 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 _step‚164 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5101582407951355 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5779752135276794 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.68498295545578 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20202568173408508 )jet/accuracy_1_of_1‚0.250853955745697 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5548245310783386 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5729166269302368 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1613832414150238 particle/accuracy_1_of_5‚1 trainer/global_step‚1999 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9896320104598999 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4777979552745819 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6558233499526978 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1558414250612259 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5102931261062622 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17608319222927094 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.993099570274353 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9870595335960388 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4276154339313507 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013578358106315136 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 particle/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9920557737350464 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27399447560310364 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46165284514427185 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17310169339179993 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 +particle/sensitivity‚0.6342698931694031 $ _timestamp‚1.784333348353756e+09 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5226345658302307 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9895392060279846 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6872269511222839 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5309880971908569 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7938397526741028 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6176835298538208 jet/accuracy_1_of_5‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1363450586795807 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8810324668884277 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9923169016838074 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6493886113166809 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17474015057086945 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46025025844573975 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9940100312232971 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5400111079216003 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5051528215408325 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.810600996017456 ,jet/accuracy_4_of_4‚0.007130124606192112 +particle/specificity‚0.8951407074928284 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872109293937683 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.25603967905044556 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16618525981903076 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08030635118484497 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2209949940443039 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16923299431800842 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 jet/accuracyEPR'² _5_of_5‚0 particle/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24271634221076965 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1613832414150238 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15503500401973724 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4814335107803345 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.985126793384552 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012643066234886646 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7350718379020691 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1594499796628952 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 _runtime‚3508.939557692 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6216179728507996 6"Purity/0FRt*SRqqt2FBt/event_purity‚0.9174844622612 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007207645568996668 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.24863281846046448 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.42824074625968933 +jet/accuracy_3_of_4‚0.07132087647914886 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5357142686843872 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15579330921173096 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5996823310852051 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5694317817687988 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11696204543113708 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.27905917167663574 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.13121142983436584 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15163739025592804 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0802433043718338 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2342168688774109 *jet/accuracy_1_of_4‚0.9639353156089783 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9944601655006409 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6071944236755371 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9916307330131531 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.027393508702516556¦îc(Ñ=ÿ Ë{ 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1613832414150238 particle/accuracy_1_of_5‚1 trainer/global_step‚1999 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9896320104598999 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4777979552745819 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6558233499526978 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1558414250612259 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5102931261062622 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17608319222927094 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.993099570274353 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9870595335960388 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4276154339313507 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013578358106315136 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 particle/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9920557737350464 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27399447560310364 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46165284514427185 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17310169339179993 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 +particle/sensitivity‚0.6342698931694031 $ _timestamp‚1.784333348353756e+09 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5226345658302307 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9895392060279846 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6872269511222839 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5309880971908569 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7938397526741028 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6176835298538208 jet/accuracy_1_of_5‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1363450586795807 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8810324668884277 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9923169016838074 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6493886113166809 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17474015057086945 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46025025844573975 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9940100312232971 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5400111079216003 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5051528215408325 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.810600996017456 ,jet/accuracy_4_of_4‚0.007130124606192112 +particle/specificity‚0.8951407074928284 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872109293937683 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.25603967905044556 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16618525981903076 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08030635118484497 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2209949940443039 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16923299431800842 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24271634221076965 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1613832414150238 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15503500401973724 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4814335107803345 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.985126793384552 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012643066234886646 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7350718379020691 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1594499796628952 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 _runtime‚3508.939557692 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6216179728507996 6"Purity/0FRt*SRqqt2FBt/event_purity‚0.9174844622612 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007207645568996668 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.24863281846046448 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.42824074625968933 +jet/accuracy_3_of_4‚0.07132087647914886 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5357142686843872 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15579330921173096 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5996823310852051 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5694317817687988 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11696204543113708 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.27905917167663574 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.13121142983436584 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15163739025592804 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0802433043718338 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2342168688774109 *jet/accuracy_1_of_4‚0.9639353156089783 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9944601655006409 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6071944236755371 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9916307330131531 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.027393508702516556 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.024294022470712662 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3878408670425415 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4480176568031311 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8063265681266785 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5285622477531433 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17329305410385132 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7152579426765442 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8448563814163208 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9744220972061157 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 *jet/accuracy_1_of_2‚0.6683337688446045 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9044681787490845 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07997293770313263 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.47544774413108826 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16918380558490753 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 jet/accuracy_3_of_5‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_4‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4982261061668396 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9979571104049683 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5499538779258728 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07456191629171371 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 )loss/val_total_loss‚9.800309887202689 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15437321364879608 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_1_of_1‚1 6validation_average_jet_accuracy‚0.2955716848373413 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9962121248245239 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5902647972106934 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2292509377002716 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15599854290485382 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0799809992313385 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.47017183899879456 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8063265681266785 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11230243742465973 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07456191629171371 particle/accuracy_2_of_2‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9965478777885437 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6097363233566284 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5762171149253845 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5505030155181885 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16987408697605133 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 /particle/accuracy_3_of_4‚0.9970238208770752 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7714150547981262 particle/accuracy_4_of_5‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4804166853427887 'particle/f_score‚0.6287062764167786 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9994933605194092 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_2_of_3‚0.4563818573951721 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6601851582527161 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 epoch‚4 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8458123803138733 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6057478785514832 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.47017183899879456 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.53125 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 +jet/accuracy_3_of_3‚0.06340016424655914 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.11449488997459412 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1208006888628006 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5799822807312012 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01777498796582222 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9936507940292358 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9946417212486267 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.47046512365341187 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11500000953674316 *jet/accuracy_1_of_3‚0.8874333500862122 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24586404860019684 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 (particle/accuracy‚0.8391191363334656 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9920565485954285 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5623694658279419 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9949482679367065 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9953996539115906 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6710695624351501 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_1_of_2‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9953996539115906 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_2_of_3‚1 /particle/accuracy_4_of_4‚0.9661495685577393 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3935185372829437 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8176871538162231 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6726835370063782 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7921391725540161 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1614629626274109 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5081256628036499 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.567078709602356 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 _step‚164 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5101582407951355 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5779752135276794 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.68498295545578 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20202568173408508 )jet/accuracy_1_of_1‚0.250853955745697 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5548245310783386 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5729166269302368 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719¤h߬ž€ ‚(:“ ¥ˆëÒûãõÇ gpu.0.gpu‚29.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚33.0 gpu.0.powerWatts‚111.353' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.270599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x.â ‚(:× ¥ˆëÒªø»É disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 123.86328125 disk.dm-0.in‚ 4178.47265625 disk.rbd0.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚23483.48095703125# proc.memory.rssMB‚ 3480.47265625 network.recv‚891631 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625# disk.dm-0.out‚23483.48095703125 network.sent‚ 3.78761e+06) disk./.usagePercent‚39.05942867761172$ disk./.usageGB‚582.0137138366699 disk.dm-1.out‚ 123.86328125 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 4178.47265625& disk.nvme0n1.out‚23483.48095703125) proc.memory.percent‚21.24311923980713 disk.sda2.out‚0 disk.rbd1.out‚9.34375 disk.sda.in‚ 0.47265625 disk.sda1.out‚0$ memory_percent‚97.68009185791016# disk.nvme0c0n1.in‚ 4178.47265625 proc.cpu.threads‚17 disk.sda1.in‚0% proc.memory.availableMB‚ 380.09375 disk.rbd0.out‚0 disk.sda.out‚ 123.86328125 disk.drbd2165.out‚0 cpu‚25.467494870945842˜¡˜‘‚ ‚(:† ´ˆëÒ‹é¶Ç gpu.0.gpu‚12.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚346.81' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.362 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$_ <ȃ ‚(:½ ´ˆëÒÙï¶É" proc.memory.rssMB‚ 2863.6015625 disk.dm-0.in‚ 4179.4375& disk.nvme0n1.out‚24462.77490234375 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚582.0175552368164' proc.memory.availableMB‚ 2291.640625* proc.memory.percent‚17.478036880493164 cpu‚25.46340768376833 network.sent‚ 3.845004e+06 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda.out‚124.125$ memory_percent‚86.01293563842773 disk.drbd2165.out‚0# disk.dm-0.out‚24462.77490234375( disk./.usagePercent‚39.0596864771987( disk.nvme0c0n1.out‚24462.77490234375 disk.sda1.out‚0 disk.dm-1.in‚ 0.47265625 disk.rbd1.out‚9.34375 disk.nvme0c0n1.in‚ 4179.4375 disk.nvme0n1.in‚ 4179.4375 disk.sda3.in‚ 0.47265625 proc.cpu.threads‚17 network.recv‚895267 disk.sda3.out‚124.125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚124.125 disk.sda.in‚ 0.47265625(uýëž„ ‚(:“ ÈëÒÑæÈ gpu.0.gpu‚22.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g¯ý ß… ‚(:Ô ÈëÒ¿ÿºÉ% disk.nvme0n1.out‚24478.0322265625 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 4238.40234375 cpu‚25.462930114633266 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda3.out‚ 125.984375' disk.nvme0c0n1.out‚24478.0322265625$ memory_percent‚86.01443767547607 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.out‚9.421875 disk.sda.out‚ 125.984375" disk.dm-0.out‚24478.0322265625 disk.sda1.in‚0* proc.memory.percent‚17.478036880493164 network.sent‚ 3.849974e+06$ disk./.usageGB‚582.0223999023438 disk.sda3.in‚ 0.47265625 disk.dm-1.in‚ 0.47265625 disk.sda.in‚ 0.47265625) proc.memory.availableMB‚ 2291.39453125" proc.memory.rssMB‚ 2863.6015625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-1.out‚ 125.984375 disk.drbd2165.in‚0 network.recv‚898694! disk.nvme0n1.in‚ 4238.40234375# disk.nvme0c0n1.in‚ 4238.40234375) disk./.usagePercent‚39.06001160676718à˜†”† ‚(:‰ ÒˆëÒ«Õ’Ç gpu.0.gpu‚93.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚311.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚62.3512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âf˜ð؇ ‚(:Í ÒˆëÒÈú”Ê* proc.memory.percent‚17.478036880493164 disk.sda3.out‚ 127.734375 disk.rbd0.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 4239.234375 disk.sda2.in‚0 disk.drbd2165.out‚0$ memory_percent‚86.01484298706055# disk./.usageGB‚582.015380859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚9.421875! disk.nvme0c0n1.in‚ 4239.234375" proc.memory.rssMB‚ 2863.6015625 disk.dm-1.in‚ 0.47265625 disk.sda3.in‚ 0.47265625& disk.nvme0n1.out‚25457.46044921875 disk.rbd0.in‚0 disk.sda.in‚ 0.47265625 disk.sda.out‚ 127.734375 proc.cpu.threads‚17 network.recv‚902148 disk.dm-1.out‚ 127.734375( disk.nvme0c0n1.out‚25457.46044921875' proc.memory.availableMB‚ 2291.328125 cpu‚25.462725936145148 network.sent‚ 3.85505e+06) disk./.usagePercent‚39.05954055290418 disk.drbd2165.in‚0 disk.dm-0.in‚ 4239.234375# disk.dm-0.out‚25457.46044921875q±ŠÃ”ˆ ‚(:‰ áˆëÒó×¢Ç gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚203.631' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.7262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tTfÖ‰ ‚(:Ë áˆëÒ²áµÊ disk.sda.out‚ 129.97265625 proc.cpu.threads‚17$ memory_percent‚86.01338863372803! disk.nvme0n1.in‚ 4240.19921875 disk.nvme0n1.out‚ 25492.15625# disk.nvme0c0n1.in‚ 4240.19921875) proc.memory.availableMB‚ 2291.56640625 disk.dm-1.out‚ 129.97265625 disk.rbd1.in‚0 disk.drbd2165.in‚0# disk./.usageGB‚582.029972076416 disk.sda1.out‚0 disk.dm-0.in‚ 4240.19921875 network.sent‚ 3.861945e+06 cpu‚25.462321465160564" disk.nvme0c0n1.out‚ 25492.15625* disk./.usagePercent‚39.060519781722654 disk.sda3.in‚ 0.47265625 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0* proc.memory.percent‚17.478036880493164 network.recv‚905524 disk.sda3.out‚ 129.97265625 disk.rbd0.out‚0 disk.rbd1.out‚9.484375" proc.memory.rssMB‚ 2863.6015625 disk.sda2.out‚0 disk.dm-0.out‚ 25492.15625 disk.sda1.in‚0 disk.rbd0.in‚0jçjJŸŠ ‚(:” ðˆëÒÄ®ûÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚292.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.480399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ' ñÖ‹ ‚(:Ë ðˆëÒŽÔüÉ disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚9.484375 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 4240.19921875 disk.drbd2165.out‚0 disk.sda3.in‚ 0.47265625$ memory_percent‚86.01493835449219 proc.cpu.threads‚17 disk.dm-1.out‚131.8125 disk.sda.in‚ 0.47265625 disk.dm-0.in‚ 4240.19921875! disk.nvme0n1.in‚ 4240.19921875 disk.rbd0.in‚0 disk.rbd0.out‚0% proc.memory.availableMB‚ 2291.3125* proc.memory.percent‚17.478036880493164 network.sent‚ 3.8686e+06 disk.dm-1.in‚ 0.47265625& disk.nvme0n1.out‚25507.45263671875 disk.sda2.in‚0" proc.memory.rssMB‚ 2863.6015625 network.recv‚916217) disk./.usagePercent‚39.05943123768706# disk.dm-0.out‚25507.45263671875 disk.sda1.out‚0( disk.nvme0c0n1.out‚25507.45263671875 cpu‚25.461791864238712$ disk./.usageGB‚582.0137519836426 disk.sda.out‚131.8125 disk.sda1.in‚0 disk.sda3.out‚131.8125mÁt\2Œ - _wandbruntime‚3596  _runtime‚3596A¢€„É Ã _runtime‚3596.709403853 _step‚165 ' lr-AdamW/pg1‚0.00047409593359707997 ' lr-AdamW/pg2‚0.00047409593359707997 trainer/global_step‚2049 % _timestamp‚1.7843334361242619e+09ãؘkÎŽ È ' lr-AdamW/pg1‚0.00047409593359707997 ' lr-AdamW/pg2‚0.00047409593359707997 trainer/global_step‚2049 % _timestamp‚1.7843334361242619e+09 _runtime‚3596.709403853 _step‚165¥¶í2 - _wandbruntime‚3596  _runtime‚3596A7=1\ éO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚588873678327808 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5261939 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5825617 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1927461341696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159279448338432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11087791 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17486701 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1928806731264 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17486701 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17757010 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1927461341696 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11087556 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5261944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17486695 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚591056341314048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚591057712877056 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚589128906145792 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17747037 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚589128906145792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2486678342656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2486645903360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1925506674442 _runtime‚3596.711019292 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17747037 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚591056341314048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588873703755776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161766494772224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚2049 _step‚166 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17486695 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161766494772224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚2049 _step‚166 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17486695 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚591056341314048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚591057712877056 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚589128906145792 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17747037 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚589128906145792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2486678342656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2486645903360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1925506674442¦†ø^2’ - _wandbruntime‚3598  _runtime‚3598Ìéåù&“ óM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17755515 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17765502 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚589406906875904 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5828375 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17755515 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚591335267079680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚589151560105984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159356934746112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3598.143887069 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚589406880702464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _step‚167 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5264487 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591078015674752 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35250416 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11092857 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1926430140800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17494901 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚589406880702464 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35250416 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17494907 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚591336638649856 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17765502 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35260409 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161845194557440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2487891720192 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5828603 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17494907 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚591076645646350 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚589406906875904 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161844794034176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚589151585533952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1929731773952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11093090TÊUV+” øM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5828603 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17494907 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚591076645646350 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚589406906875904 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161844794034176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚589151585533952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1929731773952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11093090 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17755515 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17765502 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚589406906875904 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5828375 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17755515 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚591335267079680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚589151560105984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159356934746112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3598.143887069 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚589406880702464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _step‚167 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5264487 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591078015674752 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35250416 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11092857 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1926430140800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17494901 WCDeviceStatsMonitor.on_train_batch–6Ç*Ó _end/active_bytes.large_pool.freed‚589406880702464 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35250416 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17494907 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚591336638649856 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17765502 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35260409 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161845194557440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2487891720192 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872§}942• - _wandbruntime‚3598  _runtime‚3598ÖÒa[– þ 0loss/FRt2/assignment_loss‚0.8899832842250665 /loss/FRt1/detection_loss‚0.5770744363998966 /loss/FRt2/detection_loss‚1.5382021204102785 .loss/FBt1/detection_loss‚0.533773230541715 /loss/FBt2/detection_loss‚1.0788657630148988 $loss/total_loss‚9.61800787026085 _step‚168 epoch‚5 trainer/global_step‚2049 $ _timestamp‚1.784333437609302e+09 _runtime‚3598.194421327 2loss/SRqqt1/assignment_loss‚0.5389577959186765 1loss/SRqqt1/detection_loss‚1.0181097885233765 0loss/FRt1/assignment_loss‚1.9990120360100383 2loss/SRqqt2/assignment_loss‚0.6105551175061841 1loss/FBt1/assignment_loss‚0.04713565993218928 1loss/FBt2/assignment_loss‚0.08989144988697717 1loss/SRqqt2/detection_loss‚0.6964471878915537Ƭ' — ƒ $loss/total_loss‚9.61800787026085 _step‚168 epoch‚5 trainer/global_step‚2049 $ _timestamp‚1.784333437609302e+09 _runtime‚3598.194421327 2loss/SRqqt1/assignment_loss‚0.5389577959186765 1loss/SRqqt1/detection_loss‚1.0181097885233765 0loss/FRt1/assignment_loss‚1.9990120360100383 2loss/SRqqt2/assignment_loss‚0.6105551175061841 1loss/FBt1/assignment_loss‚0.04713565993218928 1loss/FBt2/assignment_loss‚0.08989144988697717 1loss/SRqqt2/detection_loss‚0.6964471878915537 0loss/FRt2/assignment_loss‚0.8899832842250665 /loss/FRt1/detection_loss‚0.5770744363998966 /loss/FRt2/detection_loss‚1.5382021204102785 .loss/FBt1/detection_loss‚0.533773230541715 /loss/FBt2/detection_loss‚1.0788657630148988¨ô¹iᔘ ‚(:‰ ÿˆëÒ™ú¶Ç gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚288.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.6878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R" É™ ‚(:¾ ÿˆëÒûš‰Ì disk.sda1.out‚0$ memory_percent‚86.01083755493164* disk./.usagePercent‚39.059438917913084 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0# disk.dm-0.out‚25521.73291015625 cpu‚25.46136803076683$ disk./.usageGB‚582.0138664245605 disk.sda.out‚134# disk.nvme0c0n1.in‚ 4285.21484375& disk.nvme0n1.out‚25521.73291015625 disk.sda1.in‚0' proc.memory.availableMB‚ 2291.984375" proc.memory.rssMB‚ 2862.9140625 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625 disk.sda2.in‚0 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚25521.73291015625 disk.dm-0.in‚ 4285.21484375 disk.sda3.in‚ 0.47265625 disk.dm-1.out‚134 disk.rbd1.out‚9.5625 disk.sda3.out‚134* proc.memory.percent‚17.473840713500977 network.sent‚ 3.873592e+06 network.recv‚919528 disk.rbd0.out‚0! disk.nvme0n1.in‚ 4285.21484375õä]Ê”š ‚(:‰ Ž‰ëÒ…òÂÉ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚286.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.2526 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë‡Ú› ‚(:Ï Ž‰ëÒå¹õÉ disk.rbd1.in‚0) proc.memory.availableMB‚ 2291.89453125" proc.memory.rssMB‚ 2862.9296875 disk.sda1.out‚0$ memory_percent‚86.01138591766357 disk.rbd0.out‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 4285.73828125* proc.memory.percent‚17.473936080932617 disk.rbd0.in‚0 disk.sda.out‚ 134.2109375 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.nvme0n1.out‚25525.12890625 cpu‚25.461077634538835 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.in‚ 0.47265625 network.recv‚923991 disk.dm-1.in‚ 0.47265625 disk.dm-0.out‚25525.12890625! disk.nvme0n1.in‚ 4285.73828125 disk.sda1.in‚0$ disk./.usageGB‚582.0142517089844 disk.dm-1.out‚ 134.2109375 disk.rbd1.out‚9.5625% disk.nvme0c0n1.out‚25525.12890625 disk.dm-0.in‚ 4285.73828125 disk.sda.in‚ 0.47265625 network.sent‚ 3.937023e+06) disk./.usagePercent‚39.05946477467405 disk.sda3.out‚ 134.21093758Eõs”œ ‚(:‰ ‰ëÒ‚á¹Ç gpu.0.gpu‚97.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚295.976' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.1952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê#M;Î ‚(:à ‰ëÒäźÉ* disk./.usagePercent‚39.059453766350074 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.sda.out‚ 134.44921875" disk.nvme0c0n1.in‚ 4286.1796875 disk.nvme0n1.out‚ 25536.5625 disk.dm-0.in‚ 4286.1796875 cpu‚25.461175437233194 proc.cpu.threads‚17 disk.nvme0n1.in‚ 4286.1796875 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 2291.12109375" proc.memory.rssMB‚ 2863.6015625 network.sent‚ 3.945609e+06 disk.sda1.out‚0 disk.sda2.in‚0! disk.nvme0c0n1.out‚ 25536.5625 disk.dm-0.out‚ 25536.5625 network.recv‚934690# disk./.usageGB‚582.014087677002 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 134.44921875 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚86.01610660552979* proc.memory.percent‚17.478036880493164 disk.sda3.out‚ 134.44921875 disk.rbd1.out‚9.6875%„~”ž ‚(:‰ ¬‰ëÒ«¤ÌÇ gpu.0.gpu‚76.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚260.181' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0362 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢»Ö„㟠‚(:Ø ¬‰ëÒÕ–ÌÉ disk.sda1.in‚0( disk.nvme0c0n1.out‚25545.51513671875( proc.memory.availableMB‚ 2291.1171875# disk.nvme0c0n1.in‚ 4287.01171875! disk.nvme0n1.in‚ 4287.01171875& disk.nvme0n1.out‚25545.51513671875 disk.sda3.out‚ 134.72265625 disk.dm-1.out‚ 134.72265625 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0* proc.memory.percent‚17.478036880493164 disk.dm-0.in‚ 4287.01171875$ disk./.usageGB‚582.0140419006348 disk.sda3.in‚ 0.47265625 disk.sda.out‚ 134.72265625 disk.dm-1.in‚ 0.47265625 disk.drbd2165.out‚0 network.sent‚ 3.950608e+06 disk.rbd0.out‚0 disk.rbd1.out‚9.6875" proc.memory.rssMB‚ 2863.6015625 proc.cpu.threads‚17 network.recv‚938194 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0# disk.dm-0.out‚25545.51513671875# memory_percent‚86.0161304473877) disk./.usagePercent‚39.05945069425966 cpu‚25.462149491256314ûîI”  ‚(:‰ »‰ëÒÁúÄÇ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚232.436' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.4872 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö·¼å¡ ‚(:Ú »‰ëÒÜ¿ÙÉ) proc.memory.availableMB‚ 2292.01171875 cpu‚25.463183969455713$ disk./.usageGB‚582.0140380859375 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 134.94140625 proc.cpu.threads‚17 network.sent‚ 3.955554e+06 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚9.765625# disk.dm-0.out‚25552.35107421875 disk.sda1.out‚0$ memory_percent‚86.01067066192627 disk.sda2.out‚0 disk.sda1.in‚0) disk./.usagePercent‚39.05945043825213 disk.dm-1.out‚ 134.94140625 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 4342.2578125& disk.nvme0n1.out‚25552.35107421875 network.recv‚941731 disk.sda.out‚ 134.94140625 disk.dm-0.in‚ 4342.2578125 disk.nvme0n1.in‚ 4342.2578125" proc.memory.rssMB‚ 2862.9296875* proc.memory.percent‚17.473936080932617( disk.nvme0c0n1.out‚25552.35107421875çþ螢 ‚(:“ ʉëÒú¥·Ç gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚340.097' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚68.01939999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0--^°Û£ ‚(:Ð ʉëÒ†Ç×È disk.sda3.in‚ 0.47265625 disk.rbd0.out‚0) proc.memory.availableMB‚ 2291.76171875$ memory_percent‚86.01219654083252 network.recv‚944894 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 4343.22265625! disk.nvme0n1.in‚ 4343.22265625 cpu‚25.46422618904666 proc.cpu.threads‚17) disk./.usagePercent‚39.05943814989048$ disk./.usageGB‚582.0138549804688 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 135.28515625 disk.rbd1.out‚9.765625" disk.nvme0n1.out‚ 25559.7734375 disk.sda1.in‚0" proc.memory.rssMB‚ 2862.9296875* proc.memory.percent‚17.473936080932617 network.sent‚ 3.960563e+06 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625 disk.sda.out‚ 135.28515625$ disk.nvme0c0n1.out‚ 25559.7734375 disk.dm-0.in‚ 4343.22265625 disk.dm-0.out‚ 25559.7734375 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 135.28515625 disk.drbd2165.out‚0>€BÕ2¤ - _wandbruntime‚3681  _runtime‚3681 ùn:Ç¥ Á % _timestamp‚1.7843335207876709e+09 _runtime‚3681.372891467 _step‚169 & lr-AdamW/pg1‚0.0004735833890885541 & lr-AdamW/pg2‚0.0004735833890885541 trainer/global_step‚2099ÌYf“̦ Æ & lr-AdamW/pg1‚0.0004735833890885541 & lr-AdamW/pg2‚0.0004735833890885541 trainer/global_step‚2099 % _timestamp‚1.7843335207876709e+09 _runtime‚3681.372891467 _step‚169©J÷ÛŸ2§ - _wandbruntime‚3681  _runtime‚3681¯ò¶ï'¨ éO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17897001 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5389094 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17896995 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚603028942651392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1975122661888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 trainer/global_step‚2099 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚605002693750272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36069784 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚605004065313280 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚603028916477952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5965120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1971743787550 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18172789 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5964863 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18172789 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚602767767236608 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36079763 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18182762 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚165701148217344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚163154136813568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1975122661888 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5389089 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163153768722432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚604738166423836 _step‚170 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7843335207887754e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚605002693750272 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚165700745589760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3681.374004107 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2547011403776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0Kç„Ø$© îO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18182762 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11353952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2546976867328 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1973777272320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚604739536452126 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚605004065313280 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1970399187228 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2547011403776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17897001 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5389094 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17896995 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚603028942651392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1975122661888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 trainer/global_step‚2099 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚605002693750272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36069784 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚605004065313280 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚603028916477952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5965120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1971743787550 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18172789 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5964863 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18172789 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚602767767236608 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36079763 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18182762 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚165701148217344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚163154136813568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1975122661888 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5389089 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163153768722432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚604738166423836 _step‚170 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7843335207887754e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚605002693750272 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚165700745589760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3681.374004107 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17905201 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚171 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17905207 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1976047704576 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚2099 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36096461 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18181267 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚603306917208064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18191254 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2548211051520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1971322653474 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚603045649014784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843335221885824e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165779834272768 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3682.773902077 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11359456 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18191254 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1974702307840 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36086468 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚605281619515904 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1972667253908 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11359203 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36086468 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36096461 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18181267 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚603306917208064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18191254 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2548211051520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1971322653474 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚603045649014784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843335221885824e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165779834272768 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3682.773902077 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11359456 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18191254 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1974702307840 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36086468 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚605281619515904 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1972667253908 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11359203 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36086468 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17905201 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0«CÞ]Í2­ - _wandbruntime‚3682  _runtime‚3682‘0Ý® ‚ /loss/FBt2/detection_loss‚1.5506362394768407 &loss/total_loss‚10.228330593360266 trainer/global_step‚2099 % _timestamp‚1.7843335222703776e+09 1loss/SRqqt1/assignment_loss‚0.662331511090257 /loss/FRt2/detection_loss‚1.5371264415785655 1loss/SRqqt1/detection_loss‚1.2090168760765179 /loss/FBt1/detection_loss‚0.4239031938815732 _runtime‚3682.855580888 1loss/FBt1/assignment_loss‚0.03960905501058408 0loss/FRt1/assignment_loss‚1.9665487425720192 0loss/FRt2/assignment_loss‚0.9015276460704919 epoch‚5 _step‚172 2loss/SRqqt2/assignment_loss‚0.6429852283544814 2loss/FBt2/assignment_loss‚0.015505869382805121 /loss/FRt1/detection_loss‚0.5636295883147703 1loss/SRqqt2/detection_loss‚0.7155102015513597áxI– ¯ ‡ 1loss/FBt1/assignment_loss‚0.03960905501058408 0loss/FRt1/assignment_loss‚1.9665487425720192 0loss/FRt2/assignment_loss‚0.9015276460704919 epoch‚5 _step‚172 2loss/SRqqt2/assignment_loss‚0.6429852283544814 2loss/FBt2/assignment_loss‚0.015505869382805121 /loss/FRt1/detection_loss‚0.5636295883147703 1loss/SRqqt2/detection_loss‚0.7155102015513597 /loss/FBt2/detection_loss‚1.5506362394768407 &loss/total_loss‚10.228330593360266 trainer/global_step‚2099 % _timestamp‚1.7843335222703776e+09 1loss/SRqqt1/assignment_loss‚0.662331511090257 /loss/FRt2/detection_loss‚1.5371264415785655 1loss/SRqqt1/detection_loss‚1.2090168760765179 /loss/FBt1/detection_loss‚0.4239031938815732 _runtime‚3682.855580888¬ÿjµLŸ° ‚(:” Ù‰ëÒǶ÷É gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.164599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.¼Œà± ‚(:Õ Ù‰ëÒ·íüÊ$ disk./.usageGB‚582.0139846801758 disk.sda.out‚ 135.54296875 disk.sda3.out‚ 135.54296875 disk.rbd0.in‚0 disk.dm-1.out‚ 135.54296875 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd1.out‚9.828125* proc.memory.percent‚17.478036880493164 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 4344.0546875 disk.nvme0n1.in‚ 4344.0546875" proc.memory.rssMB‚ 2863.6015625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd1.in‚0# memory_percent‚86.0152006149292 cpu‚25.465212592003166% disk.nvme0n1.out‚25569.7509765625 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625) proc.memory.availableMB‚ 2291.26953125) disk./.usagePercent‚39.05944685414665 disk.dm-0.in‚ 4344.0546875" disk.dm-0.out‚25569.7509765625 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.sda2.in‚0 network.sent‚ 3.97871e+06 network.recv‚948515' disk.nvme0c0n1.out‚25569.7509765625ç'àL”² ‚(:‰ è‰ëÒýéæÈ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚114.064' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8128 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:Ú2ß³ ‚(:Ô è‰ëÒ¢ÒŠÉ disk.nvme0n1.in‚ 4344.0546875 disk.sda1.in‚0" proc.memory.rssMB‚ 2863.6015625 disk.sda.in‚ 0.47265625 disk.sda3.out‚ 137.46484375# memory_percent‚86.0175609588623 network.recv‚952975) disk./.usagePercent‚39.05966087644527 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.sda2.in‚0 disk.dm-1.out‚ 137.46484375 disk.rbd1.out‚9.828125" disk.nvme0c0n1.in‚ 4344.0546875" disk.dm-0.out‚25578.6025390625 disk.rbd0.in‚0 cpu‚25.465982736756054 disk.sda2.out‚0% disk.nvme0n1.out‚25578.6025390625( proc.memory.availableMB‚ 2290.8828125$ disk./.usageGB‚582.0171737670898 proc.cpu.threads‚17 network.sent‚ 4.02873e+06' disk.nvme0c0n1.out‚25578.6025390625 disk.dm-0.in‚ 4344.0546875* proc.memory.percent‚17.478036880493164 disk.dm-1.in‚ 0.47265625 disk.sda.out‚ 137.46484375 disk.drbd2165.out‚0 disk.rbd0.out‚0ñù”´ ‚(:‰ ÷‰ëÒùæëÇ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚118.226' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°œdäµ ‚(:Ù ÷‰ëÒ‚È÷É& disk.nvme0n1.out‚25606.21728515625 cpu‚25.46684101700191 disk.rbd1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 139.5546875) proc.memory.availableMB‚ 2291.37109375 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 4403.88671875 disk.sda3.in‚ 0.47265625 network.recv‚956684 disk.rbd1.out‚9.90625 disk.sda1.out‚0$ memory_percent‚86.01458072662354* proc.memory.percent‚17.473912239074707 network.sent‚ 4.033637e+06 disk.sda3.out‚ 139.5546875 disk.sda.in‚ 0.47265625 disk.dm-0.in‚ 4403.88671875! disk.nvme0n1.in‚ 4403.88671875 disk.sda1.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚39.06008405689938$ disk./.usageGB‚582.0234794616699 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚25606.21728515625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2862.92578125 disk.sda.out‚ 139.5546875# disk.dm-0.out‚25606.21728515625íÇ~ùž¶ ‚(:“ †ŠëÒßÂŽÈ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚286.799' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.35979999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Jü Ó· ‚(:È †ŠëÒÂíøÈ* disk./.usagePercent‚39.059626827443225 disk.sda.out‚ 141.9765625 disk.dm-1.out‚ 141.9765625 disk.rbd1.in‚0 disk.rbd1.out‚9.90625 network.sent‚ 4.038751e+06 disk.nvme0n1.in‚ 4404.71875# disk.nvme0n1.out‚25619.21484375 disk.sda2.in‚0 proc.cpu.threads‚17% disk.nvme0c0n1.out‚25619.21484375 disk.dm-0.in‚ 4404.71875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625* proc.memory.percent‚17.473936080932617$ disk./.usageGB‚582.0166664123535$ memory_percent‚91.94648265838623 cpu‚25.467718954343933 network.recv‚959856 disk.sda.in‚ 0.47265625 disk.nvme0c0n1.in‚ 4404.71875 disk.sda3.out‚ 141.9765625 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1319.48828125 disk.dm-0.out‚25619.21484375 disk.sda3.in‚ 0.47265625" proc.memory.rssMB‚ 2862.9296875amË“¸ ‚(:ˆ •ŠëÒ¬û¬È gpu.0.gpu‚97.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚289.215' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.843 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eâ·Ü¹ ‚(:Ñ •ŠëÒö¥œÉ disk.sda3.out‚ 143.51171875 disk.rbd0.out‚0 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0 proc.cpu.threads‚17 network.sent‚ 4.043609e+06 disk.nvme0n1.in‚ 4404.71875 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.out‚10.03125' disk.nvme0c0n1.out‚25630.5029296875 disk.dm-0.in‚ 4404.71875" disk.dm-0.out‚25630.5029296875 disk.sda3.in‚ 0.47265625 disk.sda.out‚ 143.51171875 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.47265625" proc.memory.rssMB‚ 2862.9296875 network.recv‚963367) disk./.usagePercent‚39.05962887550349% disk.nvme0n1.out‚25630.5029296875 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1319.48828125* proc.memory.percent‚17.473936080932617 disk.sda2.out‚0 disk.dm-1.out‚ 143.51171875$ memory_percent‚91.94648265838623 cpu‚25.468703931684963$ disk./.usageGB‚582.0166969299316 disk.nvme0c0n1.in‚ 4404.71875 disk.rbd1.in‚0¼ØR®”º ‚(:‰ ¤ŠëÒÊùöÈ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚266.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bï`Fà» ‚(:Õ ¤ŠëÒºàÛÉ disk.sda3.in‚ 0.47265625 disk.rbd1.out‚10.03125 disk.dm-1.out‚ 144.140625( proc.memory.availableMB‚ 1318.9921875* proc.memory.percent‚17.478036880493164! disk.nvme0n1.in‚ 4405.68359375 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.out‚ 144.140625 network.sent‚ 4.048612e+06 network.recv‚966843 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚25638.23291015625$ disk./.usageGB‚582.0164566040039 disk.sda3.out‚ 144.140625 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625$ memory_percent‚91.94951057434082 proc.cpu.threads‚17 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 4405.68359375# disk.dm-0.out‚25638.23291015625 cpu‚25.46954634332428 disk.sda.in‚ 0.47265625& disk.nvme0n1.out‚25638.23291015625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 4405.68359375" proc.memory.rssMB‚ 2863.6015625) disk./.usagePercent‚39.05961274702884Ô1<þ2¼ - _wandbruntime‚3765  _runtime‚3765`æ㎠¿ & lr-AdamW/pg1‚0.0004730559213862431 & lr-AdamW/pg2‚0.0004730559213862431 trainer/global_step‚2149 # _timestamp‚1.78433360479264e+09 _runtime‚3765.378048811 _step‚173{Iêʾ Ä _step‚173 & lr-AdamW/pg1‚0.0004730559213862431 & lr-AdamW/pg2‚0.0004730559213862431 trainer/global_step‚2149 # _timestamp‚1.78433360479264e+09 _runtime‚3765.378048811­ F¥´2¿ - _wandbruntime‚3765  _runtime‚3765ª™>À éO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7843336047941709e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36905836 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚618679862474034 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚618949046186496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚616928979156992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2021438592512 WJDeviceStatsMonitor.on_train_batch_start/rehYÍmÔ"quested_bytes.large_pool.current‚25427968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5516239 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚616661881573376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18307295 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18608514 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18307295 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2606764090368 _runtime‚3765.379572568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18307301 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚616928952983552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11619140 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6102640 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚616661856145408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6102896 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18598541 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2017980900658 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36915815 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚618950417749504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚618950417749504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18608514 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚616928979156992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚167028089106432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2016636300336 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚2149 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11618879 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167028457197568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2021438592512 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚618949046186496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18307301 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2020093202944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2606798626816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚174 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚185985412ªô'Á îO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2017980900658 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36915815 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚618950417749504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚618950417749504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18608514 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚616928979156992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚167028089106432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2016636300336 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚2149 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11618879 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167028457197568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2021438592512 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚618949046186496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18307301 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2020093202944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2606798626816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚174 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18598541 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7843336047941709e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36905836 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚618679862474034 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚618949046186496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚616928979156992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2021438592512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5516239 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚616661881573376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18307295 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18608514 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18307295 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2606764090368 _runtime‚3765.379572568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18307301 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚616928952983552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11619140 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6102640 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚616661856145408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6102896 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18598541®£·”2 - _wandbruntime‚3766  _runtime‚3766Åel`ù&à óM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚619229343522304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2021018238464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚617206979887104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚617206953713664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18617006 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 trainer/global_step‚2149 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36922520 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36932513 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚169713938864128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5518787 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚618958667718568 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚3766.830882031 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11624172 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36922520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2022363635200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2607960729600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2022363635200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11624434 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18315501 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18617006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2018904367016 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚617206953713664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2017559766582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18607019 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6105647 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6105390 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18315507 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18315501 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18607019 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2607995258880 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36932513 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚169713938864128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5518787 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚618958667718568 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚3766.830882031 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11624172 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36922520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2022363635200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2607960729600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.$Oøpeak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2022363635200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11624434 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18315501 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18617006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2018904367016 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚617206953713664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2017559766582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18607019 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6105647 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6105390 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18315507 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18315501 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18607019 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2607995258880 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2066409133568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6240405 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2062873413444 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37751867 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚632618818467652 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18717595 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2666617827328 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚630828989489152 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2067754523136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚632895398622720 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173569427848192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚632895398622720 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚630555970482176 _step‚178 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚632896770185728 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6240156 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11883799 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37741888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚632896770185728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2666650266624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2066409133568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19034266 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5643389 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚630828989489152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚173569027317760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚632620188495942 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18717601 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚630829015662592 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097ëHm× îO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚630829015662592 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170902409490432 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11883545 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2067754523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2066409133568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6240405 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2062873413444 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37751867 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚632618818467652 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18717595 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2666617827328 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚630828989489152 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2067754523136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚632895398622720 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173569427848192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚632895398622720 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚630555970482176 _step‚178 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚632896770185728 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6240156 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11883799 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37741888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚632896770185728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2666650266624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2066409133568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19034266 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5643389 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚630828989489152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚173569027317760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚632620188495942 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18717601 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚630829015662592 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19034266 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚2199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18717595 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11888844 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173648113674240 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173647713150976 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2063796879690 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚631107016392704 trainer/global_step‚2199 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18725801 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37758572 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18725801 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2667849684992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19042758 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚633175695958528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 _runtime‚3851.050051637 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2067334169088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37758572 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19042758 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚631106990219264 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚632897623712074 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2068679565824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2667817252864 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5645932 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18725807 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19032771 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18725807 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19032771 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚630833826832384 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚631107016392704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚633175695958528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚632898993740476 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11889083 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37768565 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6242912 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚630833852260352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚631106990219264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2068679565824 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170979895898112 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2065141480124 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚179 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5645937 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚170980263989248 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 % _timestamp‚1.7843336904645073e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2067334169088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216pzÝtþ&Ú øM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37768565 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6242912 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚630833852260352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚631106990219264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2068679565824 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170979895898112 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2065141480124 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚179 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5645937 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚170980263989248 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 % _timestamp‚1.7843336904645073e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2067334169088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6243146 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11888844 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173648113674240 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173647713150976 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2063796879690 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚631107016392704 trainer/global_step‚2199 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18725801 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37758572 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18725801 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2667849684992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19042758 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚633175695958528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 _runtime‚3851.050051637 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2067334169088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37758572 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19042758 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚631106990219264 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚632897623712074 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2068679565824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2667817252864 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5645932 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18725807 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19032771 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18725807 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19032771 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚630833826832384 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚631107016392704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚633175695958528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚632898993740476 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11889083 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0³¾”AÏ2Û - _wandbruntime‚3851  _runtime‚3851=Á‰Ü  /loss/FRt2/detection_loss‚1.6479023016211423 1loss/SRqqt1/detection_loss‚1.1322513478416427 /loss/FBt1/detection_loss‚0.5707865101880485 &loss/total_loss‚10.192495023277685 epoch‚5 0loss/FRt1/assignment_loss‚1.9039279472481858 2loss/SRqqt1/assignment_loss‚0.7002062445702837 _step‚180 0loss/FRt2/assignment_loss‚0.9494399727045835 /loss/FRt1/detection_loss‚0.5482340936351312 1loss/SRqqt2/detection_loss‚0.6854325237453474 /loss/FBt2/detection_loss‚1.1573648056464976 % _timestamp‚1.7843336905368576e+09 trainer/global_step‚2199 _runtime‚3851.122084061 2loss/SRqqt2/assignment_loss‚0.5922073116842401 1loss/FBt1/assignment_loss‚0.07557613323953655 0loss/FBt2/assignment_loss‚0.2291658311530447oÿÑ Ý † 1loss/SRqqt1/detection_loss‚1.1322513478416427 /loss/FBt1/detection_loss‚0.5707865101880485 &loss/total_loss‚10.192495023277685 epoch‚5 0loss/FRt1/assignment_loss‚1.9039279472481858 2loss/SRqqt1/assignment_loss‚0.7002062445702837 _step‚180 0loss/FRt2/assignment_loss‚0.9494399727045835 /loss/FRt1/detection_loss‚0.5482340936351312 1loss/SRqqt2/detection_loss‚0.6854325237453474 /loss/FBt2/detection_loss‚1.1573648056464976 % _timestamp‚1.7843336905368576e+09 trainer/global_step‚2199 _runtime‚3851.122084061 2loss/SRqqt2/assignment_loss‚0.5922073116842401 1loss/FBt1/assignment_loss‚0.07557613323953655 0loss/FBt2/assignment_loss‚0.2291658311530447 /loss/FRt2/detection_loss‚1.6479023016211423´¯êênžÞ ‚(:“ þŠëÒ¡ýêÈ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚330.528' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.10560000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚-¨Õß ‚(:Ê þŠëÒ—¤ÏÉ disk.sda3.out‚ 154.953125 cpu‚25.47515072356133 disk.drbd2165.in‚0* proc.memory.percent‚17.473936080932617 disk.rbd1.in‚0 disk.sda3.in‚ 0.47265625 proc.cpu.threads‚17$ memory_percent‚91.94488525390625# proc.memory.availableMB‚1319.75 network.recv‚987917 disk.nvme0n1.in‚ 4530.9296875" proc.memory.rssMB‚ 2862.9296875" disk.nvme0c0n1.in‚ 4530.9296875 disk.dm-0.in‚ 4530.9296875 network.sent‚ 4.135694e+06# disk.dm-0.out‚25727.30810546875 disk.sda2.out‚0 disk.sda.out‚ 154.953125 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚582.0180015563965 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 154.953125 disk.rbd1.out‚10.25 disk.sda.in‚ 0.47265625& disk.nvme0n1.out‚25727.30810546875) disk./.usagePercent‚39.05971643008021 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚25727.30810546875 disk.sda1.in‚0 disk.sda1.out‚0k~6Ÿà ‚(:” ‹ëÒâ“ÿÈ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚276.811' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.362199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á2hÙá ‚(:Î ‹ëÒžÎŠË disk.sda2.in‚0 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚25743.86181640625 disk.rbd0.out‚0$ memory_percent‚91.94390773773193" proc.memory.rssMB‚ 2862.9296875 network.recv‚992388) disk./.usagePercent‚39.05987208266102# disk./.usageGB‚582.020320892334 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 4530.96875 disk.rbd0.in‚0 network.sent‚ 4.198675e+06 disk.dm-1.out‚ 156.2421875 disk.rbd1.out‚10.3125 disk.dm-0.in‚ 4530.96875 disk.sda3.out‚ 156.2421875) proc.memory.availableMB‚ 1319.91015625 disk.rbd1.in‚0( disk.nvme0c0n1.out‚25743.86181640625 disk.nvme0n1.in‚ 4530.96875 cpu‚25.47599650783055 disk.sda.out‚ 156.2421875# disk.dm-0.out‚25743.86181640625 disk.sda1.out‚0* proc.memory.percent‚17.473936080932617 proc.cpu.threads‚17 disk.dm-1.in‚ 0.47265625 disk.sda3.in‚ 0.47265625¸“+W”â ‚(:‰ œ‹ëÒæâïÆ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚297.228' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.4456 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ«»æã ‚(:Û œ‹ëÒ¤ýÕÉ disk.rbd1.in‚0 disk.dm-0.in‚ 4531.80078125$ disk./.usageGB‚582.0217781066895 disk.dm-1.out‚ 158.99609375 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2863.6015625* proc.memory.percent‚17.478036880493164 proc.cpu.threads‚17! disk.nvme0n1.in‚ 4531.80078125 disk.sda3.in‚ 0.47265625# disk.nvme0c0n1.in‚ 4531.80078125( disk.nvme0c0n1.out‚25768.51025390625$ memory_percent‚91.94848537445068) proc.memory.availableMB‚ 1319.16015625 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2165.in‚0& disk.nvme0n1.out‚25768.51025390625 cpu‚25.476491635142587 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.out‚10.3125# disk.dm-0.out‚25768.51025390625 disk.sda1.out‚0 disk.sda.in‚ 0.47265625 disk.sda.out‚ 158.99609375 network.sent‚ 4.203739e+06( disk./.usagePercent‚39.0599698775391 disk.sda3.out‚ 158.99609375 network.recv‚995554 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625œ}R#žä ‚(:“ «‹ëÒŠÊ§É gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚282.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.45099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åTœ˜âå ‚(:× «‹ëÒ•þÂÊ$ disk./.usageGB‚582.0144195556641 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625 disk.dm-0.in‚ 4532.765625$ memory_percent‚91.94557666778564 disk.sda3.in‚ 0.47265625* proc.memory.percent‚17.478036880493164 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 4532.765625) disk./.usagePercent‚39.05947603900555 disk.dm-1.out‚ 162.30078125 disk.sda.out‚ 162.30078125) proc.memory.availableMB‚ 1319.63671875 network.sent‚ 4.210644e+06 disk.nvme0n1.in‚ 4532.765625 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 10.390625 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 162.30078125 disk.drbd2165.in‚0 network.recv‚998802& disk.nvme0n1.out‚25788.45458984375# disk.dm-0.out‚25788.45458984375" proc.memory.rssMB‚ 2863.6015625 disk.sda.in‚ 0.47265625( disk.nvme0c0n1.out‚25788.45458984375 cpu‚25.47746163420584 proc.cpu.threads‚17ÞÜø+”æ ‚(:‰ º‹ëÒÖÍûÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚262.589' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5178 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë¬ßç ‚(:Ô º‹ëÒÄå€Ê disk.sda2.in‚0 network.recv‚ 1.002342e+06) disk./.usagePercent‚39.05948960740487 disk.rbd1.out‚ 10.390625! disk.nvme0n1.in‚ 4587.89453125 disk.rbd1.in‚0 disk.sda3.out‚162.875 disk.rbd0.in‚0 disk.dm-1.out‚162.875# disk.nvme0c0n1.in‚ 4587.89453125 cpu‚25.478315612301003 network.sent‚ 4.213815e+06 disk.sda2.out‚0 disk.drbd2165.out‚0* proc.memory.percent‚17.478036880493164# disk.dm-0.out‚25791.58642578125& disk.nvme0n1.out‚25791.58642578125 disk.sda1.in‚0 disk.sda.out‚162.875 disk.dm-0.in‚ 4587.89453125 disk.dm-1.in‚ 0.47265625$ memory_percent‚91.94865226745605 disk.sda3.in‚ 0.47265625 disk.rbd0.out‚0 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0$ disk./.usageGB‚582.0146217346191 disk.sda1.out‚0( disk.nvme0c0n1.out‚25791.58642578125( proc.memory.availableMB‚ 1319.1328125" proc.memory.rssMB‚ 2863.6015625 proc.cpu.threads‚17›U?”è ‚(:‰ É‹ëÒ¾¯Ç gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚281.966' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%éSFé ‚(:Ø É‹ëÒó™¦É$ memory_percent‚91.94879531860352" proc.memory.rssMB‚ 2863.6015625 cpu‚25.479229146788036 proc.cpu.threads‚17% disk.nvme0n1.out‚25803.2919921875 disk.dm-0.in‚ 4588.7265625 disk.drbd2165.out‚0$ disk./.usageGB‚582.0145378112793 disk.nvme0n1.in‚ 4588.7265625 dis+EIòk.drbd2165.in‚0' proc.memory.availableMB‚ 1319.109375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 network.sent‚ 4.21868e+06) disk./.usagePercent‚39.05948397523912" disk.dm-0.out‚25803.2919921875 network.recv‚ 1.005743e+06 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 163.5234375 disk.sda.in‚ 0.47265625 disk.sda.out‚ 163.5234375* proc.memory.percent‚17.478036880493164 disk.rbd1.in‚0 disk.rbd1.out‚ 10.515625" disk.nvme0c0n1.in‚ 4588.7265625' disk.nvme0c0n1.out‚25803.2919921875 disk.sda2.out‚0 disk.dm-1.out‚ 163.5234375LQÅß2ê - _wandbruntime‚3934  _runtime‚3934>Ö#^Éë à _runtime‚3934.154769439 _step‚181 ' lr-AdamW/pg1‚0.00047195635286963764 ' lr-AdamW/pg2‚0.00047195635286963764 trainer/global_step‚2249 % _timestamp‚1.7843337735696151e+09 PhÎì È ' lr-AdamW/pg1‚0.00047195635286963764 ' lr-AdamW/pg2‚0.00047195635286963764 trainer/global_step‚2249 % _timestamp‚1.7843337735696151e+09 _runtime‚3934.154769439 _step‚181µ·¯Å2í - _wandbruntime‚3934  _runtime‚3934Sï'î éO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843337735708857e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚646843122621952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚644729025994752 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚644729025994752 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19450045 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177503893945344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2249 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2112725064192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2114070453760 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19127901 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚646559144489560 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2110455126874 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38577940 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174777097965568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19127895 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6378134 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚644729052168192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2112725064192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚646841751058944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚646841751058944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19460018 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12148911 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5770539 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12148673 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174776729874432 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚182 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19127901 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚646843122621952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚646560514517850 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175ŽÃnFô'ï îO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6378134 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚644729052168192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2112725064192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚646841751058944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚646841751058944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19460018 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12148911 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5770539 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12148673 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174776729874432 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚182 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19127901 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚646843122621952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚646560514517850 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843337735708857e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚646843122621952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚644729025994752 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚644729025994752 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19450045 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177503893945344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2249 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2112725064192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2114070453760 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19127901 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚646559144489560 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2110455126874 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38577940 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174777097965568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19127895 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995¶˜³<12ð - _wandbruntime‚3935  _runtime‚3935ê:ˆQù&ñ óM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38604617 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19136107 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174854584373248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12154196 _step‚183 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2113650099712 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 trainer/global_step‚2249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2113650099712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12153968 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38594624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174854216282112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚645007052898304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2114995496448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19136107 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚645007026724864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38604617 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚646837949733982 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19458523 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2728001607680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19458523 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚645007052898304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19136101 % _timestamp‚1.7843337749788492e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38594624 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177582185457664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚647122048394752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19468510 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5773082 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19468510 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚645007026724864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2114995496448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚644727915741184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2110033992798 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38604617 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚646837949733982 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19458523 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2728001607680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStats9ãúäê"Monitor.on_train_batch_end/active.small_pool.freed‚19458523 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚645007052898304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19136101 % _timestamp‚1.7843337749788492e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38594624 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177582185457664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚647122048394752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19468510 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5773082 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19468510 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚645007026724864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2114995496448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚644727915741184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2110033992798 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174854584373248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12154196 _step‚183 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2113650099712 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 trainer/global_step‚2249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2113650099712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12153968 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38594624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174854216282112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚645007052898304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2114995496448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19136107 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚645007026724864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39423971 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚658629088673792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19538201 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2786964624384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚658629062500352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚186 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚2299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6516576 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚660500840539758 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2155347639660 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚660788103495168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚658629062500352 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2156692239982 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19875797 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19538195 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚660788103495168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12414270 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚660499470511468 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5897689 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6516344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12414033 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2786997063680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 % _timestamp‚1.7843338578828106e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181438415413248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚660789475058176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19538195 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚660789475058176 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5897694 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39413992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19875797 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2160386384384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2159040994816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚658629088673792 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2159040994816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19885770O¶fÝ… íO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2159040994816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚658629088673792 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2159040994816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19885770 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚ 4018.46814954 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚658344122871808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19885770 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2160386384384 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181438014882816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚178651050258432 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19538201 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚178651418349568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39423971 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚658629088673792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19538201 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2786964624384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚658629062500352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚186 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚2299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6516576 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚660500840539758 ZHDeviceStatsMoniti7:or.on_train_batch_start/requested_bytes.small_pool.freed‚ 2155347639660 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚660788103495168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚658629062500352 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2156692239982 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19875797 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19538195 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚660788103495168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12414270 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚660499470511468 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5897689 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6516344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12414033 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2786997063680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 % _timestamp‚1.7843338578828106e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181438415413248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚660789475058176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19538195 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚660789475058176 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5897694 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39413992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19875797 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2160386384384º:cÄ¢2† - _wandbruntime‚4019  _runtime‚4019ÿ ù&‡ óM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6519106 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚178728536666112 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚658907089403904 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2159966030336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178728904757248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19894262 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12419338 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5900237 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚658907063230464 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2161311427072 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5900232 % _timestamp‚1.7843338593084373e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚226 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚661068400830976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19894262 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19546407 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚658622030077952 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19546407 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19546401 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2157615706340 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12419569 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181516702092288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2159966030336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39440669 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39440669 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚661067029260800 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12419569 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181516702092288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2159966030336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39440669 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39440669 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚661067029260800 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚658907089403904 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2159966030336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178728904757248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19894262 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12419338 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5900237 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚658907063230464 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2161311427072 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5900232 % _timestamp‚1.7843338593084373e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚226 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚661068400830976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19894262 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736»ßÏI2‰ - _wandbruntime‚4019  _runtime‚4019t뢊 þ /loss/FRt1/assignment_loss‚1.971363352654615 0loss/FRt2/assignment_loss‚0.8631168840345272 2loss/SRqqt1/assignment_loss‚0.6774818502374537 2loss/SRqqt2/assignment_loss‚0.6064787238258046 /loss/FRt2/detection_loss‚1.5125200114451534 1loss/SRqqt1/detection_loss‚1.0064960200171678 0loss/SRqqt2/detection_loss‚0.666415898287867 % _timestamp‚1.7843338593798494e+09 0loss/FBt1/assignment_loss‚0.0635249073494919 /loss/FRt1/detection_loss‚0.5430891555677846 %loss/total_loss‚9.254199576758225 epoch‚5 _runtime‚4019.964892806 /loss/FBt1/detection_loss‚0.5230157834726076 /loss/FBt2/detection_loss‚0.8003970200736676 1loss/FBt2/assignment_loss‚0.02029996979208517 _step‚188 trainer/global_step‚2299É“1 ‹ ƒ % _timestamp‚1.7843338593798494e+09 0loss/FBt1/assignment_loss‚0.0635249073494919 /loss/FRt1/detection_loss‚0.5430891555677846 %loss/total_loss‚9.254199576758225 epoch‚5 _runtime‚4019.964892806 /loss/FBt1/detection_loss‚0.5230157834726076 /loss/FBt2/detection_loss‚0.8003970200736676 1loss/FBt2/assignment_loss‚0.02029996979208517 _step‚188 trainer/global_step‚2299 /loss/FRt1/assignment_loss‚1.971363352654615 0loss/FRt2/assignment_loss‚0.8631168840345272 2loss/SRqqt1/assignment_loss‚0.6774818502374537 2loss/SRqqt2/assignment_loss‚0.6064787238258046 /loss/FRt2/detection_loss‚1.5125200114451534 1loss/SRqqt1/detection_loss‚1.0064960200171678 0loss/SRqqt2/detection_loss‚0.666415898287867¼ŒÙ’ÞŒ ‚(:“ £ŒëÒ½Ö·Ç gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚285.199' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.03980000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!Þïê ‚(:ß £ŒëÒ½óõÉ" disk.nvme0c0n1.in‚ 4647.8515625( disk.nvme0c0n1.out‚25846.80419921875 disk.dm-1.out‚ 165.3515625 disk.dm-0.in‚ 4647.8515625 disk.sda1.in‚0 disk.sda3.out‚ 165.3515625 disk.rbd0.in‚0 disk.sda2.in‚0 network.sent‚ 4.306802e+06& disk.nvme0n1.out‚25846.80419921875 disk.dm-1.in‚ 0.47265625 disk.drbd2165.out‚0 disk.sda3.in‚ 0.47265625 disk.sda.in‚ 0.47265625* disk./.usagePercent‚39.059499335691164 disk.rbd0.out‚0" proc.memory.rssMB‚ 2863.6015625$ disk./.usageGB‚582.0147666931152 disk.sda2.out‚0 disk.rbd1.out‚ 10.796875 disk.sda.out‚ 165.3515625# disk.dm-0.out‚25846.80419921875 disk.nvme0n1.in‚ 4647.8515625 disk.rbd1.in‚0* proc.memory.percent‚17.478036880493164 proc.cpu.threads‚17 network.recv‚ 1.027679e+06 disk.sda1.out‚0 disk.drbd2165.in‚0$ memory_percent‚91.94576740264893) proc.memory.availableMB‚ 1319.60546875 cpu‚25.483885650996225 ä.;”Ž ‚(:‰ ²ŒëÒ˜åÈÉ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.843' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1686 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è1Êã ‚(:Ø ²ŒëÒ¥éãÊ disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 165.53125 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚25849.8212890625" proc.memory.rssMB‚ 2863.6015625 network.sent‚ 4.371825e+06% disk.nvme0n1.out‚25849.8212890625 disk.sda1.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚582.0146713256836! disk.nvme0n1.in‚ 4883.16015625 disk.sda3.in‚ 0.47265625 disk.drbd2165.out‚0 disk.dm-0.in‚ 4883.16015625 network.recv‚ 1.032534e+06 disk.sda.in‚ 0.47265625# disk.nvme0c0n1.in‚ 4883.16015625* proc.memory.percent‚17.478036880493164 disk.sda3.out‚ 165.53125 disk.dm-1.in‚ 0.47265625 disk.sda.out‚ 165.53125) proc.memory.availableMB‚ 1319.30078125) disk./.usagePercent‚39.05949293550282 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 10.796875 cpu‚25.484857482413148 proc.cpu.threads‚17 disk.rbd1.in‚0" disk.dm-0.out‚25849.8212890625$ memory_percent‚91.94762706756592 O˜” ‚(:‰ ÁŒëÒãµÓÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚264.249' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.8498 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡ƒ…lÙ‘ ‚(:Î ÁŒëÒ£ ÚÉ disk.sda1.in‚0$ disk./.usageGB‚582.0147399902344 disk.dm-1.out‚ 165.83203125 disk.rbd1.in‚0 disk.rbd1.out‚ 10.859375 disk.sda3.out‚ 165.83203125 disk.sda3.in‚ 0.47265625 network.sent‚ 4.374789e+06 network.recv‚ 1.035911e+06 disk.dm-0.in‚4884.125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4884.125) disk./.usagePercent‚39.05949754363843 disk.nvme0n1.in‚4884.125" proc.memory.rssMB‚ 2863.6015625 proc.cpu.threads‚17$ memory_percent‚91.95148944854736) proc.memory.availableMB‚ 1318.66796875* proc.memory.percent‚17.478036880493164 cpu‚25.48560199407866 disk.sda.in‚ 0.47265625& disk.nvme0c0n1.out‚25859.001953125! disk.dm-0.out‚25859.001953125$ disk.nvme0n1.out‚25859.001953125 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625 disk.sda.out‚ 165.83203125 disk.drbd2165.in‚0 disk.drbd2165.out‚07ì8”’ ‚(:‰ ÐŒëÒô¹áÈ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚275.567' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.1134 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jð“ ‚(:á ÐŒëÒ‡þºÉ disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 165.9453125 cpu‚25.486452563020052 disk.sda.out‚ 165.9453125( disk.nvme0c0n1.out‚25866.76513671875! disk.nvme0n1.in‚ 4884.95703125 disk.sda.in‚ 0.47265625 disk.dm-0.in‚ 4884.95703125 disk.sda2.out‚0 disk.rbd1.out‚ 10.859375& disk.nvme0n1.out‚25866.76513671875 disk.sda1.out‚0$ memory_percent‚91.95055961608887 network.sent‚[Ð3þÓ 4.379862e+06 network.recv‚ 1.039221e+06# disk.dm-0.out‚25866.76513671875 disk.sda1.in‚0( proc.memory.availableMB‚ 1318.8203125" proc.memory.rssMB‚ 2863.6015625* proc.memory.percent‚17.478036880493164* disk./.usagePercent‚39.059458118478155 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.sda3.out‚ 165.9453125$ disk./.usageGB‚582.0141525268555# disk.nvme0c0n1.in‚ 4884.95703125 proc.cpu.threads‚17—:òö”” ‚(:‰ ߌëÒñï¡Ç gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s©,ç• ‚(:Ü ߌëÒ‰˜’É disk.sda3.in‚ 0.47265625 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0 disk.sda.out‚ 166.33203125$ disk.nvme0n1.out‚25872.931640625 disk.sda1.in‚0 network.sent‚ 4.384748e+06& disk.nvme0c0n1.out‚25872.931640625* disk./.usagePercent‚39.059473478930215 disk.rbd0.in‚0 network.recv‚ 1.042629e+06 cpu‚25.48729788846129) proc.memory.availableMB‚ 1318.78515625 disk.sda1.out‚0 disk.dm-1.out‚ 166.33203125 disk.dm-0.in‚ 4884.95703125! disk.dm-0.out‚25872.931640625 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.out‚10.9375! disk.nvme0n1.in‚ 4884.95703125$ disk./.usageGB‚582.0143814086914 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 4884.95703125 disk.sda2.in‚0 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2863.6015625 proc.cpu.threads‚17* proc.memory.percent‚17.478036880493164$ memory_percent‚91.95077419281006 disk.sda3.out‚ 166.33203125Ep#fž– ‚(:“ îŒëÒè×¹È gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚325.701' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.14020000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªLôFì— ‚(:á îŒëÒ뤬É network.sent‚ 4.389721e+06 disk.rbd0.in‚0( disk.nvme0c0n1.out‚25881.33154296875 disk.sda2.out‚0& disk.nvme0n1.out‚25881.33154296875 disk.sda3.in‚ 0.47265625 disk.dm-1.out‚ 166.49609375$ memory_percent‚91.95072650909424" proc.memory.rssMB‚ 2863.6015625 disk.rbd1.out‚10.9375 disk.sda.in‚ 0.47265625) proc.memory.availableMB‚ 1318.79296875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 4983.84765625 cpu‚25.488069705051764( disk./.usagePercent‚39.0594704068398$ disk./.usageGB‚582.0143356323242 disk.sda3.out‚ 166.49609375 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 disk.sda.out‚ 166.49609375# disk.nvme0c0n1.in‚ 4983.84765625* proc.memory.percent‚17.478036880493164 network.recv‚ 1.045957e+06 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 4983.84765625# disk.dm-0.out‚25881.33154296875¹ÉE)2˜ - _wandbruntime‚4103  _runtime‚4103vJ Ä™ ¾ % lr-AdamW/pg1‚0.000470797510535652 % lr-AdamW/pg2‚0.000470797510535652 trainer/global_step‚2349 % _timestamp‚1.7843339424678953e+09 _runtime‚ 4103.05308355 _step‚189ð, Éš à % _timestamp‚1.7843339424678953e+09 _runtime‚ 4103.05308355 _step‚189 % lr-AdamW/pg1‚0.000470797510535652 % lr-AdamW/pg2‚0.000470797510535652 trainer/global_step‚2349½;Üc2› - _wandbruntime‚4103  _runtime‚4103F9Âï'œ éO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20311522 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 % _timestamp‚1.7843339424694502e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚672529125179392 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚672529099005952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2206702315008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6024844 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20301549 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19948501 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 trainer/global_step‚2349 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19948495 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12678764 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚674735827494400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚672238211780608 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2201584752768 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40250044 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚672529125179392 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20301549 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6654158 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6024839 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2205356925440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12679002 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚674439796533376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19948501 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2847211242496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _step‚190 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19948495 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚185372549445632 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚185372949976064 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2847178803200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚674734455931392 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚674735827494400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40260023 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6653925 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚674441166561666 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚672238237208576 _runtime‚4103.054966294 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182525370642432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20311522 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2206702315008 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12679002 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚674439796533376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19948501 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2847211242496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _step‚190 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19948495 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚185372549445632 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚185372949976064 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2847178803200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚674734455931392 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚674735827494400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40260023 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6653925 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚674441166561666 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚672238237208576 _runtime‚4103.054966294 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182525370642432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20311522 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2206702315008 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0¾aWÃ2ž - _wandbruntime‚4104  _runtime‚4104ä/•Ç Ÿ óM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182603225141248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12684277 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2848380432384 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843339439050274e+09 _step‚191 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40266728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚674718601777798 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2207627357696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19956707 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚672516118986752 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40276721 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚674719971806200 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19956701 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚675014753267200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19956707 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12684047 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2206281960960 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6027387 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2202508219014 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6656665 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19956701 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40276721 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚672807125909504 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20320014 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6027382 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E?DeviceStatsMonitor.on_train_batch_end/active.(—wž2large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚672516093558784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚675014753267200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚185451237482496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2848412864512 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2203852819448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚672807099736064 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20310027 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚185451638005760í òŸþ&  øM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40276721 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚672807125909504 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20320014 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6027382 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚672516093558784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚675014753267200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚185451237482496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2848412864512 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2203852819448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚672807099736064 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20310027 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚185451638005760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182603225141248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12684277 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2848380432384 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843339439050274e+09 _step‚191 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40266728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚674718601777798 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2207627357696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19956707 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚672516118986752 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40276721 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚674719971806200 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19956701 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚675014753267200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19956707 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12684047 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2206281960960 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6027387 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2202508219014 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6656665 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19956701 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820¿BÚk 2¡ - _wandbruntime‚4104  _runtime‚4104 ŒÕe¢ ÿ 0loss/FRt2/assignment_loss‚0.9212357284950798 2loss/FBt1/assignment_loss‚0.035816934965447286 0loss/FBt2/assignment_loss‚0.0718520494145013 epoch‚5 0loss/FRt1/assignment_loss‚1.9548272839277097 /loss/FRt1/detection_loss‚0.5770378847624634 /loss/FBt1/detection_loss‚0.5765412311723574 /loss/FBt2/detection_loss‚0.6430396262941689 %loss/total_loss‚9.242359906314398 trainer/global_step‚2349 1loss/SRqqt1/assignment_loss‚0.588614299624227 /loss/FRt2/detection_loss‚1.5692745699338486 1loss/SRqqt1/detection_loss‚1.0144002927038818 1loss/SRqqt2/detection_loss‚0.6933550069328851 _runtime‚4104.548369044 _step‚192 2loss/SRqqt2/assignment_loss‚0.5963649980878275 $ _timestamp‚1.784333943963238e+09hÌö £ „ 2loss/SRqqt2/assignment_loss‚0.5963649980878275 $ _timestamp‚1.784333943963238e+09 0loss/FRt2/assignment_loss‚0.9212357284950798 2loss/FBt1/assignment_loss‚0.035816934965447286 0loss/FBt2/assignment_loss‚0.0718520494145013 epoch‚5 0loss/FRt1/assignment_loss‚1.9548272839277097 /loss/FRt1/detection_loss‚0.5770378847624634 /loss/FBt1/detection_loss‚0.5765412311723574 /loss/FBt2/detection_loss‚0.6430396262941689 %loss/total_loss‚9.242359906314398 trainer/global_step‚2349 1loss/SRqqt1/assignment_loss‚0.588614299624227 /loss/FRt2/detection_loss‚1.5692745699338486 1loss/SRqqt1/detection_loss‚1.0144002927038818 1loss/SRqqt2/detection_loss‚0.6933550069328851 _runtime‚4104.548369044 _step‚192À©Ä5P¤ ‚(:… ýŒëÒ嫾Ç gpu.0.gpu‚9.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð^þí¥ ‚(:â ýŒëÒ¦êªÉ disk.sda.in‚ 0.47265625 disk.dm-0.in‚ 4983.84765625 network.recv‚ 1.049643e+06 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 4983.84765625 disk.drbd2165.in‚0 cpu‚25.48883851808808 disk.sda3.in‚ 0.47265625& disk.nvme0n1.out‚25888.73974609375 network.sent‚ 4.394789e+06 disk.rbd1.out‚11.0625 disk.sda1.in‚0" proc.memory.rssMB‚ 2863.6015625* disk./.usagePercent‚39.059467590756924 disk.sda.out‚ 166.72265625 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚25888.73974609375# disk.dm-0.out‚25888.73974609375) proc.memory.availableMB‚ 1319.01171875* proc.memory.percent‚17.478036880493164 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625! disk.nvme0n1.in‚ 4983.84765625 disk.sda3.out‚ 166.72265625 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 166.72265625 disk.sda1.out‚0$ memory_percent‚91.94939136505127 proc.cpu.threads‚17$ disk./.usageGB‚582.0142936706543 disk.rbd1.in‚0|áÕÉ”¦ ‚(:‰ ŒëÒ‰™ŸÇ gpu.0.gpu‚76.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚276.632' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3264 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I}àVä§ ‚(:Ù ŒëÒËÑÊ disk.sda3.in‚ 0.47265625) disk./.usagePercent‚39.05946195859117$ disk./.usageGB‚582.0142097473145' disk.nvme0c0n1.out‚25896.4306640625 disk.dm-0.in‚ 4984.6796875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚11.0625 disk.sda.in‚ 0.47265625 disk.sda.out‚ 166.9609375 disk.drbd2165.out‚0$ memory_percent‚91.94953441619873 network.sent‚ 4.460034e+06 disk.sda1.out‚0 disk.dm-1.out‚ 166.9609375) proc.memory.availableMB‚ 1318.98828125 network.recv‚ 1.054741e+06" disk.dm-0.out‚25896.4306640625 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 4984.6796875% disk.nvme0n1.out‚25896.4306640625* proc.memory.percent‚17.478036880493164 disk.sda2.out‚0" proc.memory.rssMB‚ 2863.6015625 cpu‚25.489651890044126" disk.nvme0c0n1.in‚ 4984.6796875 disk.sda3.out‚ 166.9609375 proc.cpu.threads‚17 disk.sda2.in‚05·ý>”¨ ‚(:‰ ›ëÒëÏùÆ gpu.0.gpu‚73.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚151.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.2046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤¦Bïì© ‚(:á ›ëÒˆ°”É disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625 disk.sda1.out‚0* proc.memory.percent‚17.478036880493164 disk.rbd1.in‚0' disk.nvme0c0n1.out‚25907.5908203125! disk.nvme0n1.in‚ 4985.64453125) proc.memory.availableMB‚ 1318.72265625$ disk./.usageGB‚582.0143280029297 disk.sda2.in‚0 disk.dm-1.out‚ 167.57421875# disk.nvme0c0n1.in‚ 4985.64453125 network.recv‚ 1.058371e+06) disk./.usagePercent‚39.05946989482473 disk.sda.in‚ 0.47265625% disk.nvme0n1.out‚25907.5908203125$ memory_percent‚91.95115566253662 cpu‚25.490181683409986 disk.sda2.out‚0 disk.sda.out‚ 167.57421875 disk.sda3.out‚ 167.57421875 disk.sda1.in‚0 disk.rbd1.out‚ 11.140625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 4985.64453125 network.sent‚ 4.463163e+06" disk.dm-0.out‚25907.5908203125 disk.sda3.in‚ 0.47265625" proc.memory.rssMB‚ 2863.6015625 proc.cpu.threads‚17¶ÝÃK’ª ‚(:‡ ªëÒ¾˜ªÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚267.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0  ïã« ‚(:Ø ªëÒ¹â¼É disk.rbd1.in‚0* proc.memory.percent‚17.478036880493164 network.recv‚ 1.061547e+06$ disk./.usageGB‚582.0149688720703 disk.dm-1.in‚ 0.47265625 cpu‚25.490990256645688 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 5059.7890625 disk.sda1.out‚0 disk.rbd0.in‚0) disk./.usagePercent‚39.05951290409048 disk.sda2.in‚0 disk.drbd2165.in‚0" disk.dm-0.out‚25911.6201171875 disk.sda2.out‚0 disk.sda.out‚ 167.859375' disk.nvme0c0n1.out‚25911.6201171875 disk.dm-0.in‚ 5059.7890625 disk.dm-1.out‚ 167.859375 disk.drbd2165.out‚0 disk.sda3.in‚ 0.47265625 disk.nvme0n1.in‚ 5059.7890625% disk.nvme0n1.out‚25911.6201171875 disk.rbd0.out‚0) proc.memory.availableMB‚ 1318.72265625 disk.rbd1.out‚ 11.140625 disk.sda.in‚ 0.47265625 disk.sda1.in‚0 disk.sda3.out‚ 167.859375$ memory_percent‚91.95115566253662" proc.memory.rssMB‚ 2863.6015625 network.sent‚ 4.468021e+06ì•úО¬ ‚(:“ ¹ëÒõÚÇ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚282.166' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.43319999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_›ê­ ‚(:ß ¹ëÒþÈÛÉ network.recv‚ 1.065134e+06 disk.sda3.out‚ 168.1484375 disk.sda3.in‚ 0.47265625 disk.sda2.in‚0# disk.dm-0.out‚25922.21630859375 disk.dm-0.in‚ 5060.62109375 cpu‚25.491794099101842 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 168.1484375 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 5060.62109375( disk.nvme0c0n1.out‚25922.21630859375( proc.memory.availableMB‚ 1318.4921875& disk.nvme0n1.out‚25922.21630859375 disk.rbd1.out‚ 11.203125 disk.sda.in‚ 0.47265625 disk.sda.out‚ 168.1484375 network.sent‚ 4.473039e+06 disk.sda2.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.05951930427884# disk./.usageGB‚582.015064239502! disk.nvme0n1.in‚ 5060.62109375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0$ memory_percent‚91.95256233215332* proc.memory.percent‚17.478036880493164 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2863.6015625”ÎÃß”® ‚(:‰ ÈëÒŠÅ¬È gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚234.349' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.8698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20727301 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚686429161684992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20358801 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12943882 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚686429161684992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚186400059117568 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20727301 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6151989 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚688680808367616 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚186399691026432 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41096075 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚688380122555284 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41086096 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20737274 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚688680808367616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚686429135511552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚4187.562946969 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12943623 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2253018245632 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2247821865876 trainer/global_step‚2399 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20358795 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6791888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6791634 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189306719622144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚688381492583574 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚686132300689408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20358801 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚688682179930624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚688682179930624 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6151994 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2249166466198 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2251672856064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚686132326117376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20358795 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _step‚194 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚688680808367616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚686429135511552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚4187.562946969 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12943623 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2253018245632 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2247821865876 trainer/global_step‚2399 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20358795 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6791888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6791634 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189306719622144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚688381492583574 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚686132300689408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20358801 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚688682179930624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚688682179930624 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6151994 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2249166466198 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2251672856064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚686132326117376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20358795 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _step‚194 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20727301 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚686429161684992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20358801 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12943882 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚686429161684992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚186400059117568 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20727301 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6151989 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚688680808367616 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚186399691026432 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872ª>u­2¶ - _wandbruntime‚4189  _runtime‚4189S¬E=ù&· óM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41102780 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41112773 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2248745332122 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚686707162415104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚189385395477504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20745766 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20735779 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚688658927799706 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20735779 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚2399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20367001 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 _step‚195 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41102780 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚686707136241664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186477177434112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843340286124635e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41112773 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12949145 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2252597891584 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12948907 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6154537 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2250089932556 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6794608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚688660297828108 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2253943288320 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6794375 _runtime‚4189.197995758 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20367007 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚688959734133248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚688961105703424 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2252597891584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚686410182467584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192…Þ—Nþ&¸ øM M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 _step‚195 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41102780 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚686707136241664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186477177434112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843340286124635e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41112773 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12949145 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2252597891584 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12948907 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6154537 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2250089932556 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6794608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚688660297828108 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2253943288320 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6794375 _runtime‚4189.197995758 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20367007 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚688959734133248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚688961105703424 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2252597891584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚686410182467584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41102780 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41112773 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2248745332122 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚686707162415104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚189385395477504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20745766 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20735779 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚688658927799706 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20735779 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚2399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20367001 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 E&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.32251593470573425 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7507747411727905 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1459672600030899 particle/accuracy_1_of_3‚1 /particle/accuracy_4_of_4‚0.9729472398757935 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2718123495578766 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9819908142089844 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.19718758761882782 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01977410539984703 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9076329469680786 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02807447500526905 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 6validation_average_jet_accuracy‚0.2861892879009247 jet/accuracy_4_of_5‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965230822563171 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2572385370731354 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6943803429603577 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.30000001192092896 *jet/accuracy_1_of_3‚0.8835927844047546 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4823032021522522 'particle/f_score‚0.6224573850631714 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4895783066749573 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2745170295238495 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.19050057232379913 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4886201322078705 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3322512209415436 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6171875 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1811005175113678 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 )jet/accuracy_1_of_2‚0.667849600315094 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.560880720615387 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8374906182289124 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.37783291935920715 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2506812512874603 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4004629850387573 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25795307755470276 /particle/accuracy_3_of_3‚0.9994933605194092 particle/accuracy_4_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20122277736663818 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6273783445358276 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4289979040622711 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.41698646545410156 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8398905396461487 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5716176629066467 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25499463081359863 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.41155529022216797 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9941746592521667 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚2399 % _timestamp‚1.7843340576098845e+09 _runtime‚4218.195904681 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 jet/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9634513854980469 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3785240650177002 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7097001671791077 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9864512085914612 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5839591026306152 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967845678329468 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7916666865348816 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.39058995246887207 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.09248528629541397 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.668291449546814 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6298342347145081 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2037438005208969 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.37317606806755066 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3791842460632324 jet/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2633466124534607 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976301789283752 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9874768853187561 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2828904390335083 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2054179310798645 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.31102272868156433 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3824183940887451 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5840657353401184 ,jet/accuracy_4_of_4‚0.011098379269242287 +particle/sensitivity‚0.6115283370018005 jet/accuracy_5_of_5‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 *jet/accuracy_2_of_4‚0.7385727763175964 +particle/specificity‚0.9033846855163574 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_1‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.353641539812088 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.28489357233047485 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.36330947279930115 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7097001671791077 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1459672600030899 'particle/accuracy‚0.840704083442688 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2824275493621826 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20901723206043243 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6484035849571228 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5674063563346863 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1955895870923996 ,"Purity/2FRt0SRqqt1FBt/event_purity‚0.375 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8587002754211426 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9186739921569824 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44212961196899414 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.3719707727432251 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5952350497245789 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.24610108137130737 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.09097069501876831 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.605704128742218 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4883711636066437 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9817256927490234 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 +jet/accuracy_1_of_1‚0.23751063644886017 *jet/accuracy_2_of_3‚0.4793751537799835 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.054582614451646805 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9959007501602173 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3047577738761902 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9924865961074829 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.029597481712698936 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.503333330154419 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1803203523159027 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5577276945114136 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9909769892692566 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +jet/accuracy_2_of_2‚0.11628928780555725 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.09427926689386368 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8191989064216614 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5523200035095215 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9940598011016846 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9914049506187439 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4064056873321533 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7364515662193298 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.023856027051806450ws i; ¿{ 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9817256927490234 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 +jet/accuracy_1_of_1‚0.23751063644886017 *jet/accuracy_2_of_3‚0.4793751537799835 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.054582614451646805 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9959007501602173 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3047577738761902 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9924865961074829 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.029597481712698936 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.503333330154419 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1803203523159027 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5577276945114136 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9909769892692566 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +jet/accuracy_2_of_2‚0.11628928780555725 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.09427926689386368 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8191989064216614 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5523200035095215 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9940598011016846 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9914049506187439 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4064056873321533 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7364515662193298 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2426319122314453 (loss/val_total_loss‚9.39669153146792 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.985907256603241 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7916666865348816 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3414405286312103 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.24610108137130737 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.07061431556940079 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8363568186759949 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3551023304462433 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.21156242489814758 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19227877259254456 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49247586727142334 *jet/accuracy_1_of_4‚0.9526292085647583 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.39930862188339233 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8531633019447327 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9845833778381348 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.37759795784950256 particle/accuracy_2_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8985834717750549 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_3‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 epoch‚5 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6584820747375488 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 _step‚197 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.23715761303901672 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9907057881355286 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8453893065452576 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5446524024009705 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22243104875087738 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.3788731098175049 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.32251593470573425 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.980201005935669 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3355298638343811 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18211179971694946 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5993944406509399 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2504727244377136 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.36546236276626587 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03855077549815178 /particle/accuracy_3_of_4‚0.9970238208770752 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.018413718789815903 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9925240874290466 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967845678329468 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5252633690834045 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.42067161202430725 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.32251593470573425 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7507747411727905 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1459672600030899 particle/accuracy_1_of_3‚1 /particle/accuracy_4_of_4‚0.9729472398757935 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2718123495578766 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9819908142089844 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.19718758761882782 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01977410539984703 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9076329469680786 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02807447500526905 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 6validation_average_jet_accuracy‚0.2861892879009247 jet/accuracy_4_of_5‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965230822563171 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2572385370731354 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6943803429603577 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.30000001192092896 *jet/accuracy_1_of_3‚0.8835927844047546 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4823032021522522 'particle/f_score‚0.6224573850631714 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4895783066749573 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2745170295238495 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.19050057232379913 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4886201322078705 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3322512209415436 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6171875 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1811005175113678 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 )jet/accuracy_1_of_2‚0.667849600315094 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.560880720615387 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8374906182289124 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.37783291935920715 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2506812512874603 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4004629850387573 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25795307755470276 /particle/accuracy_3_of_3‚0.9994933605194092 particle/accuracy_4_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20122277736663818 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6273783445358276 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4289979040622711 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.41698646545410156 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8398905396461487 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5716176629066467 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25499463081359863 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.41155529022216797 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9941746592521667 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚2399 % _timestamp‚1.7843340576098845e+09 _runtime‚4218.195904681 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 jet/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9634513854980469 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3785240650177002 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7097001671791077 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9864512085914612 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5839591026306152 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967845678329468 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7916666865348816 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.39058995246887207 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.09248528629541397 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.668291449546814 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6298342347145081 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2037438005208969 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.37317606806755066 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3791842460632324 jet/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2633466124534607 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976301789283752 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9874768853187561 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2828904390335083 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2054179310798645 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.31102272868156433 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3824183940887451 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5840657353401184 ,jet/accuracy_4_of_4‚0.011098379269242287 +particle/sensitivity‚0.6115283370018005 jet/accuracy_5_of_5‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 *jet/accuracy_2_of_4‚0.7385727763175964 +particle/specificity‚0.9033846855163574 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_1‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.353641539812088 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.28489357233047485 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.36330947279930115 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7097001671791077 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1459672600030899 'particle/accuracy‚0.840704083442688 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2824275493621826 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20901723206043243 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6484035849571228 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5674063563346863 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1955895870923996 ,"Purity/2FRt0SRqqt1FBt/event_purity‚0.375 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8587002754211426 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9186739921569824 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44212961196899414 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.3719707727432251 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5952350497245789 & Purit5…\y/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.24610108137130737 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.09097069501876831 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.605704128742218 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4883711636066437 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516Å6në”à ‚(:‰ õëÒ›ÇÈ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚257.521' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5042 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&¦„bÞÄ ‚(:Ó õëÒîÛÉ) proc.memory.percent‚17.58279800415039! disk.nvme0c0n1.in‚ 5131.046875! proc.memory.rssMB‚ 2880.765625 network.recv‚ 1.079592e+06) disk./.usagePercent‚39.05952621648227 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0$ memory_percent‚86.08474731445312 network.sent‚ 4.550134e+06 disk.nvme0n1.in‚ 5131.046875& disk.nvme0n1.out‚27954.83154296875 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 169.1953125 disk.dm-0.in‚ 5131.046875 disk.rbd1.out‚11.34375 proc.cpu.threads‚17$ disk./.usageGB‚582.0151672363281# disk.dm-0.out‚27954.83154296875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda.out‚ 169.1953125 disk.sda1.in‚0 disk.sda2.in‚0$ proc.memory.availableMB‚2279.875 disk.sda3.in‚ 0.47265625 disk.rbd0.out‚0 disk.rbd1.in‚0 cpu‚25.507890264576154( disk.nvme0c0n1.out‚27954.83154296875 disk.sda3.out‚ 169.1953125 disk.drbd2165.out‚0—!s¸”Å ‚(:‰ „ŽëÒǪÛÇ gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚257.222' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.4444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04¥>âÆ ‚(:× „ŽëÒü†âÉ network.sent‚ 4.607626e+06 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0 disk.rbd1.out‚11.34375 cpu‚25.50868155879352$ disk./.usageGB‚582.0152053833008 disk.nvme0n1.in‚ 5131.046875 disk.sda2.out‚0) proc.memory.availableMB‚ 2279.88671875 proc.cpu.threads‚17( disk.nvme0c0n1.out‚27958.05224609375 disk.sda1.out‚0 disk.dm-1.out‚ 169.3359375 disk.sda.in‚ 0.47265625 network.recv‚ 1.083853e+06& disk.nvme0n1.out‚27958.05224609375 disk.dm-0.in‚ 5131.046875 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 169.3359375) proc.memory.percent‚17.58279800415039! disk.nvme0c0n1.in‚ 5131.046875 disk.rbd0.in‚0 disk.rbd0.out‚0# memory_percent‚86.0846757888794* disk./.usagePercent‚39.059528776557606 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.dm-0.out‚27958.05224609375 disk.sda2.in‚0 disk.sda.out‚ 169.3359375! proc.memory.rssMB‚ 2880.765625¬¯Ç ‚(:„ “ŽëÒ³þÊÈ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚337.0' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.4 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ŸIäÈ ‚(:Ù “ŽëÒÇûïÈ disk.dm-1.in‚ 0.47265625# disk.nvme0c0n1.in‚ 5131.87890625) disk./.usagePercent‚39.05952416842199 disk.drbd2165.in‚0" disk./.usageGB‚582.01513671875 disk.sda.in‚ 0.47265625 network.sent‚ 4.612584e+06 disk.sda3.in‚ 0.47265625$ memory_percent‚86.08458042144775 disk.sda2.in‚0 disk.sda.out‚ 169.5859375% disk.nvme0n1.out‚27967.9521484375) proc.memory.availableMB‚ 2279.90234375 disk.dm-1.out‚ 169.5859375 disk.rbd1.out‚ 11.484375 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.58279800415039 cpu‚25.50947914001709 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 5131.87890625! disk.nvme0n1.in‚ 5131.87890625 disk.sda3.out‚ 169.5859375 disk.rbd0.in‚0! proc.memory.rssMB‚ 2880.765625' disk.nvme0c0n1.out‚27967.9521484375" disk.dm-0.out‚27967.9521484375 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚ 1.087457e+06”D¥É ‚(:’ ¢ŽëÒËíýÈ gpu.0.gpu‚18.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚328.402' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.68039999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄƽëÊ ‚(:à ¢ŽëÒ®’¹É( disk.nvme0c0n1.out‚27975.73974609375 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 169.99609375 disk.dm-1.in‚ 0.47265625 proc.cpu.threads‚17 disk.sda.in‚ 0.47265625 disk.sda.out‚ 169.99609375 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 11.484375 cpu‚25.510310950107723* disk./.usagePercent‚39.059519048271305# disk.dm-0.out‚27975.73974609375& disk.nvme0n1.out‚27975.73974609375 disk.sda2.in‚0 network.sent‚ 4.617591e+06 network.recv‚ 1.090673e+06 disk.nvme0n1.in‚ 5226.2734375 disk.sda2.out‚0 disk.dm-1.out‚ 169.99609375) proc.memory.availableMB‚ 2280.08203125! proc.memory.rssMB‚ 2880.765625$ disk./.usageGB‚582.0150604248047" disk.nvme0c0n1.in‚ 5226.2734375 disk.rbd0.in‚0$ memory_percent‚86.08348369598389) proc.memory.percent‚17.58279800415039 disk.drbd2165.out‚0 disk.dm-0.in‚ 5226.2734375 disk.drbd2165.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 º Ë ‚(:… ±ŽëÒ»¦È gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.òç&çÌ ‚(:Ü ±ŽëÒÅÇ¡É disk.dm-1.in‚ 0.47265625 disk.sda.in‚ 0.47265625 disk.nvme0n1.in‚ 5226.2734375 disk.sda1.out‚0 disk.sda3.out‚ 170.21484375 disk.dm-1.out‚ 170.21484375$ memory_percent‚86.08157634735107 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.578983306884766 cpu‚25.510856691147346 disk.sda3.in‚ 0.47265625 disk.rbd1.out‚ 11.546875' disk.nvme0c0n1.out‚27980.4658203125% disk.nvme0n1.out‚27980.4658203125 proc.cpu.threads‚17 network.recv‚ 1.094025e+06" disk.nvme0c0n1.in‚ 5226.2734375 disk.rbd0.out‚0 network.sent‚ 4.622546e+06 disk.rbd1.in‚0" disk.dm-0.out‚27980.4658203125! proc.memory.rssMB‚ 2880.140625) disk./.usagePercent‚39.05951930427884 disk.drbd2165.in‚0 disk.drbd2165.out‚0) proc.memory.availableMB‚ 2280.39453125# disk./.usageGB‚582.015064239502 disk.sda.out‚ 170.21484375 disk.dm-0.in‚ 5226.2734375 disk.rbd0.in‚0öÉr“Í ‚(:ˆ ÀŽëÒ˜õËÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.715' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.343 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑçÎ ‚(:Ü ÀŽëÒˆÒÉ! disk.nvme0n1.in‚ 5227.10546875 disk.drbd2165.in‚0 network.recv‚ 1.097468e+06 disk.dm-1.in‚ 0.47265625$ memory_percent‚86.08160018920898 disk.sda2.out‚0 disk.dm-1.out‚ 170.3984375 disk.drbd2165.out‚0 disk.dm-0.in‚ 5227.10546875 cpu‚25.51162268854228$ disk./.usageGB‚582.0151596069336& disk.nvme0n1.out‚27987.78076171875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 170.3984375 disk.rbd1.out‚ 11.546875 disk.sda.out‚ 170.3984375! proc.memory.rssMB‚ 2880.140625 proc.cpu.threads‚17( disk./.usagePercent‚39.0595257044672# disk.dm-0.out‚27987.78076171875* proc.memory.percent‚17.578983306884766 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.sda2.in‚0 disk.sda.in‚ 0.47265625 network.sent‚ 4.627625e+06# disk.nvme0c0n1.in‚ 5227.10546875( disk.nvme0c0n1.out‚27987.78076171875' proc.memory.availableMB‚ 2280.390625v–ÈÒ2Ï - _wandbruntime‚4306  _runtime‚4306QܨbÉÐ Ã ' lr-AdamW/pg2‚0.00046957969210061637 trainer/global_step‚2449 % _timestamp‚1.7843341461870549e+09 _runtime‚4306.772110583 _step‚198 ' lr-AdamW/pg1‚0.00046957969210061637 ™2ÊÎÑ È ' lr-AdamW/pg2‚0.00046957969210061637 trainer/global_step‚2449 % _timestamp‚1.7843341461870549e+09 _runtime‚4306.772110583 _step‚198 ' lr-AdamW/pg1‚0.00046957969210061637Ɖ¾;42Ò - _wandbruntime‚4306  _runtime‚4306Íeå¡ï'Ó éO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13251747 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚706152186641242 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193297084577280 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚704157153452032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2974558292480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚706462356416000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚703852298530816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6963690 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚706462356416000 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2301232710748 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚704157179625472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚190322491748352 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚703852323958784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20901101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6288057 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚2449 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚704157153452032 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚190322123657216 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21212040 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42123120 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚706460984852992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2303831400960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968=|šô'Ô îO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21222013 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193296681949696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚706460984852992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2299888110426 % _timestamp‚1.7843341461883197e+09 _runtime‚4306.774831529 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚704157179625472 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2305176790528 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6288062 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20901101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6288057 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚2449 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚704157153452032 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚190322123657216 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21212040 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42123120 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚706460984852992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2303831400960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13251747 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚706152186641242 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193297084577280 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚704157153452032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2974558292480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚706462356416000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚703852298530816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6963690 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚706462356416000 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2301232710748 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚704157179625472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚190322491748352 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚703852323958784 B- Ö òM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6966709 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2302156177106 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2304756436480 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚704130205736960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _runtime‚ 4308.19896155 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13257065 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚704435180355584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20909313 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚190399610064896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20909307 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21220518 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20909307 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42129825 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2306101833216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2304756436480 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2975793622528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚706432361914066 trainer/global_step‚2449 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21230505 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42139818 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42129825 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42139818 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13257314 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2306101833216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2300811576672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚706430991885664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚706739910618624 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 % _timestamp‚1.7843341476134622e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6966465 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20909313 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6290605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193375771778560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2975759093248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21230505 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42139818 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42129825 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42139818 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13257314 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2306101833216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2300811576672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚706430991885664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚706739910618624 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 % _timestamp‚1.7843341476134622e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6966465 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20909313 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6290605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193375771778560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2975759093248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2302156177106 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2304756436480 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚704130205736960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _runtime‚ 4308.19896155 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13257065 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚704435180355584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20909313 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚190399610064896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20909307 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176Èí]®2Ø - _wandbruntime‚4308  _runtime‚4308ÇÏÙ  1loss/SRqqt2/detection_loss‚0.8079174859244581 /loss/FBt2/detection_loss‚1.1388364436570555 _runtime‚ 4308.27835456 _step‚201 0loss/FRt2/assignment_loss‚0.9682216373245411 1loss/FBt2/assignment_loss‚0.03651129935053543 /loss/FRt1/detection_loss‚0.5712788959508507 1loss/SRqqt1/detection_loss‚1.1432240665079612 trainer/global_step‚2449 0loss/FRt1/assignment_loss‚2.0667436771005034 /loss/FBt1/detection_loss‚0.5940069163359863 epoch‚6 2loss/SRqqt1/assignment_loss‚0.5673214010159043 2loss/FBt1/assignment_loss‚0.016399390125535776 .loss/FRt2/detection_loss‚1.537870565525145 &loss/total_loss‚10.001184721288347 % _timestamp‚1.7843341476933472e+09 2loss/SRqqt2/assignment_loss‚0.5528529424698688:½µx Ú † _step‚201 0loss/FRt2/assignment_loss‚0.9682216373245411 1loss/FBt2/assignment_loss‚0.03651129935053543 /loss/FRt1/detection_loss‚0.5712788959508507 1loss/SRqqt1/detection_loss‚1.1432240665079612 trainer/global_step‚2449 0loss/FRt1/assignment_loss‚2.0667436771005034 /loss/FBt1/detection_loss‚0.5940069163359863 epoch‚6 2loss/SRqqt1/assignment_loss‚0.5673214010159043 2loss/FBt1/assignment_loss‚0.016399390125535776 .loss/FRt2/detection_loss‚1.537870565525145 &loss/total_loss‚10.001184721288347 % _timestamp‚1.7843341476933472e+09 2loss/SRqqt2/assignment_loss‚0.5528529424698688 1loss/SRqqt2/detection_loss‚0.8079174859244581 /loss/FBt2/detection_loss‚1.1388364436570555 _runtime‚ 4308.27835456ɤþàèžÛ ‚(:“ ÏŽëÒŽØ‰É gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚273.684' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.73680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zÞÜ ‚(:Ó ÏŽëÒ®µ°É% disk.nvme0n1.out‚27993.1904296875 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 5227.10546875! proc.memory.rssMB‚ 2880.765625! disk.nvme0n1.in‚ 5227.10546875 disk.sda.out‚ 170.65625 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚27993.1904296875 cpu‚25.512448869746304$ disk./.usageGB‚582.0148506164551 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 170.65625 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0 network.sent‚ 4.63263e+06* disk./.usagePercent‚39.059504967856924 disk.dm-0.in‚ 5227.10546875) proc.memory.availableMB‚ 2280.01953125 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 170.65625) proc.memory.percent‚17.58279800415039 disk.rbd1.out‚11.625 disk.sda.in‚ 0.47265625 disk.sda1.out‚0" disk.dm-0.out‚27993.1904296875$ memory_percent‚86.08386516571045 proc.cpu.threads‚17 network.recv‚ 1.100877e+06s@/˜’Ý ‚(:‡ ÞŽëÒüºžÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚261.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ­–ßÞ ‚(:Ô ÞŽëÒÉÝèÈ disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 170.9140625 disk.nvme0n1.in‚ 5295.7578125$ memory_percent‚92.01664924621582 disk.sda.out‚ 170.9140625 disk.sda3.in‚ 0.47265625 proc.cpu.threads‚17$ disk./.usageGB‚582.0146598815918 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 5295.7578125 disk.sda1.in‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2880.765625 network.sent‚ 4.695665e+06 disk.rbd1.out‚11.625 disk.rbd0.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.58279800415039 cpu‚25.51321561479808) disk./.usagePercent‚39.05949216748021 disk.rbd1.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚28000.4052734375" disk.dm-0.out‚28000.4052734375 disk.sda3.out‚ 170.9140625 disk.dm-0.in‚ 5295.7578125% disk.nvme0n1.out‚28000.4052734375 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0( proc.memory.availableMB‚ 1307.9921875 network.recv‚ 1.105267e+06oŒVŸß ‚(:” íŽëÒâÕ×Ç gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚269.302' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.860400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`°éà ‚(:Þ íŽëÒò±ãÉ disk.rbd1.in‚0$ memory_percent‚92.01364517211914' proc.memory.availableMB‚ 1308.484375 disk.sda1.out‚0 disk.sda3.out‚ 171.14453125 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 5296.58984375 disk.sda1.in‚0 disk.dm-1.out‚ 171.14453125 disk.rbd1.out‚11.6875( disk.nvme0c0n1.out‚28011.01416015625& disk.nvme0n1.out‚28011.01416015625 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 network.recv‚ 1.108939e+06$ disk./.usageGB‚582.0146713256836! disk.nvme0n1.in‚ 5296.58984375 disk.rbd0.out‚0! proc.memory.rssMB‚ 2880.140625 cpu‚25.51395857423478) disk./.usagePercent‚39.05949293550282# disk.dm-0.out‚28011.01416015625 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 disk.sda.in‚ 0.47265625 disk.sda.out‚ 171.14453125 disk.drbd2165.in‚0 disk.dm-0.in‚ 5296.58984375* proc.memory.percent‚17.578983306884766 proc.cpu.threads‚17 network.sent‚ 4.700743e+06$ oNžá ‚(:“ üŽëÒôó»Ç gpu.0.gpu‚89.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚299.167' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.83339999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07ùXìâ ‚(:á üŽëÒûäË disk.dm-1.in‚ 0.47265625& disk.nvme0n1.out‚28014.22802734375( proc.memory.availableMB‚ 1308.4921875$ disk./.usageGB‚582.0146446228027 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.out‚11.6875 disk.sda.out‚ 171.41796875( disk.nvme0c0n1.out‚28014.22802734375 network.recv‚ 1.112131e+06 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 171.41796875 disk.rbd0.out‚0 disk.dm-1.out‚ 171.41796875 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 5296.58984375 cpu‚25.514690664215127 disk.sda2.in‚0! proc.memory.rssMB‚ 2880.140625* disk./.usagePercent‚39.059491143450074 disk.drbd2165.in‚0 disk.dm-0.in‚ 5296.58984375# disk.nvme0c0n1.in‚ 5296.58984375# disk.dm-0.out‚28014.22802734375$ memory_percent‚92.01359748840332 network.sent‚ 4.705636e+06 disk.sda1.in‚0 disk.sda.in‚ 0.47265625 proc.cpu.threads‚17 disk.rbd1.in‚0* proc.memory.percent‚17.578983306884766óџٔ㠂(:‰ ‹ëÒ«¿ÈÈ gpu.0.gpu‚85.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚295.631' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.1262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Eh7Uìä ‚(:á ‹ëÒ …É$ disk./.usageGB‚582.0159683227539 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625" disk.nvme0c0n1.in‚ 5297.5546875) proc.memory.availableMB‚ 1308.48828125! proc.memory.rssMB‚ 2880.140625* proc.memory.percent‚17.578983306884766 network.recv‚ 1.115297e+06* disk./.usagePercent‚39.059579978064455 disk.rbd1.in‚0( disk.nvme0c0n1.out‚28023.68603515625 proc.cpu.threads‚17 network.sent‚ 4.710619e+06 disk.sda.in‚ 0.47265625 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 171.66015625 disk.rbd0.in‚0 disk.rbd1.out‚ 11.765625 disk.sda.out‚ 171.66015625 cpu‚25.515403630826803 disk.dm-1.out‚ 171.66015625 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 5297.5546875& disk.nvme0n1.out‚28023.68603515625 disk.sda2.in‚0# disk.dm-0.out‚28023.68603515625 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 5297.5546875 disk.rbd0.out‚0$ memory_percent‚92.01362133026123%*H2å - _wandbruntime‚4391  _runtime‚4391^Z5Åæ ¿ trainer/global_step‚2499 $ _timestamp‚1.784334230683537e+09 _runtime‚ 4391.26850726 _step‚202 & lr-AdamW/pg1‚0.0004689487640267053 & lr-AdamW/pg2‚0.0004689487640267053µ´¨Êç Ä _step‚202 & lr-AdamW/pg1‚0.0004689487640267053 & lr-AdamW/pg2‚0.0004689487640267053 trainer/global_step‚2499 $ _timestamp‚1.784334230683537e+09 _runtime‚ 4391.26850726ʧëf62è - _wandbruntime‚4391  _runtime‚4391T3Zõé èO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 % _timestamp‚1.7843342306846206e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21637792 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2347469823856 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21311401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21311407 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42949193 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21637792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21311401 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6415212 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21311407 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2350147331584 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚197230820511232 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3034413103616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚720092512663150 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚720408708852224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7103131 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197231225235968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194196812132352 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2351492721152 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42959172 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194196444041216 trainer/global_step‚2499 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚720408708852224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚718057189957632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13518626 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2350147331584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13518338 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2346125223534 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595Ç“&&ù008 _step‚203 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚720407337289216 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚720407337289216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3034376470016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚717746387439616 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21647765 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 _runtime‚ 4391.27296724 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚718057189957632 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21647765 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚718057216131072 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2351492721152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7103414 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640¤–þÅó'ê íO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21311401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21311407 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42949193 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21637792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21311401 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6415212 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21311407 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2350147331584 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚197230820511232 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3034413103616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚720092512663150 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚720408708852224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7103131 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197231225235968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194196812132352 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2351492721152 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42959172 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194196444041216 trainer/global_step‚2499 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚720408708852224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚718057189957632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13518626 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2350147331584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13518338 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2346125223534 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _step‚203 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚720407337289216 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚720407337289216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3034376470016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚717746387439616 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21647765 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 _runtime‚ 4391.27296724 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚718057189957632 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21647765 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚718057216131072 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2351492721152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7103414 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 % _timestamp‚1.7843342306846206e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21637792 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2347469823856 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820Ë£€_J2ë - _wandbruntime‚4392  _runtime‚4392 >Ãgø&ì òM TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21656257 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚718335190687744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42965877 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197309899705856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197309494988288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42965877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚718335216861184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7105888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚720687634625024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3035601165824 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚718335216861184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2351072367104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13523638 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚4392.732059984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚720687634625024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21646270 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194274298540032 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42975870 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21319607 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3035564539392 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚720371317907572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2347048689780 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2352417763840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚2499 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21656257 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21319613 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2348393290214 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6417755 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42975870 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21319613 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚718335190687744 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21319607 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2352417763840 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚720686263054848 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7106156 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194273930448896 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚718024269217792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13523911 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21656257 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21319613 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2348393290214 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6417755 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42975870 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21319613 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚718335190687744 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21319607 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2352417763840 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚720686263054848 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7106156 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194273930448896 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚718024269217792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13523911 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42965877 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197309899705856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197309494988288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42965877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚718335216861184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7105888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚720687634625024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3035601165824 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚718335216861184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2351072367104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13523638 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚4392.732059984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚720687634625024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21646270 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_by×_í6tes.large_pool.allocated‚194274298540032 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42975870 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21319607 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3035564539392 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚720371317907572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2347048689780 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2352417763840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Ì?ô•2î - _wandbruntime‚4392  _runtime‚4392zsaÖï ÿ 2loss/SRqqt1/assignment_loss‚0.6306409825404846 2loss/FBt1/assignment_loss‚0.032899935543779686 -loss/FBt2/detection_loss‚1.14826099774135 %loss/total_loss‚9.969814898260843 1loss/FBt2/assignment_loss‚0.09121350097600255 /loss/FRt1/detection_loss‚0.5690246345039626 /loss/FRt2/detection_loss‚1.5804280070730645 1loss/SRqqt1/detection_loss‚1.0229071415960789 epoch‚6 trainer/global_step‚2499 _step‚205 0loss/FRt1/assignment_loss‚1.9449374993010808 0loss/FRt2/assignment_loss‚1.1582510069632166 2loss/SRqqt2/assignment_loss‚0.6327629057213998 /loss/FBt1/detection_loss‚0.4745479924571035 1loss/SRqqt2/detection_loss‚0.6839402938433188 $ _timestamp‚1.784334232189607e+09 _runtime‚4392.774818246uÌ*? ð „ 1loss/SRqqt1/detection_loss‚1.0229071415960789 epoch‚6 trainer/global_step‚2499 _step‚205 0loss/FRt1/assignment_loss‚1.9449374993010808 0loss/FRt2/assignment_loss‚1.1582510069632166 2loss/SRqqt2/assignment_loss‚0.6327629057213998 /loss/FBt1/detection_loss‚0.4745479924571035 1loss/SRqqt2/detection_loss‚0.6839402938433188 $ _timestamp‚1.784334232189607e+09 _runtime‚4392.774818246 2loss/SRqqt1/assignment_loss‚0.6306409825404846 2loss/FBt1/assignment_loss‚0.032899935543779686 -loss/FBt2/detection_loss‚1.14826099774135 %loss/total_loss‚9.969814898260843 1loss/FBt2/assignment_loss‚0.09121350097600255 /loss/FRt1/detection_loss‚0.5690246345039626 /loss/FRt2/detection_loss‚1.5804280070730645Í mÊ™Ÿñ ‚(:” šëÒ´ŒÈ gpu.0.gpu‚85.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚243.212' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.642399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…â½uæò ‚(:Û šëÒƒÜÕÉ network.sent‚ 4.715598e+06& disk.nvme0n1.out‚28031.43408203125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 171.87109375 disk.sda3.out‚ 171.87109375 disk.dm-0.in‚ 5381.57421875 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625) proc.memory.percent‚17.58279800415039 cpu‚25.516080037020643# disk./.usageGB‚582.016674041748 disk.rbd0.out‚0 disk.sda.in‚ 0.47265625 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚92.01812744140625# proc.memory.availableMB‚1307.75! proc.memory.rssMB‚ 2880.765625) disk./.usagePercent‚39.05962733945829# disk.nvme0c0n1.in‚ 5381.57421875# disk.dm-0.out‚28031.43408203125! disk.nvme0n1.in‚ 5381.57421875 disk.sda3.in‚ 0.47265625 network.recv‚ 1.118435e+06 disk.rbd1.out‚ 11.765625( disk.nvme0c0n1.out‚28031.43408203125 disk.sda1.in‚0 disk.sda.out‚ 171.87109375º|6žó ‚(:“ ©ëÒ¦¾’Ç gpu.0.gpu‚86.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚333.764' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.75280000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"mÚ—åô ‚(:Ú ©ëÒ‘ΗÊ disk.sda.in‚ 0.47265625 disk.sda.out‚ 172.23046875# disk.nvme0c0n1.in‚ 5381.57421875 network.recv‚ 1.12317e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚28041.0673828125! disk.nvme0n1.in‚ 5381.57421875 disk.rbd0.in‚0' proc.memory.availableMB‚ 1308.328125 proc.cpu.threads‚17 disk.sda2.in‚0% disk.nvme0n1.out‚28041.0673828125$ memory_percent‚92.01459884643555 cpu‚25.516808624058438 disk.sda3.out‚ 172.23046875 disk.rbd0.out‚0 network.sent‚ 4.778533e+06) disk./.usagePercent‚39.05962989953363 disk.dm-1.in‚ 0.47265625 disk.rbd1.out‚ 11.828125 disk.dm-0.in‚ 5381.57421875 proc.memory.rssMB‚2880.125 disk.sda2.out‚0 disk.dm-1.out‚ 172.23046875" disk.dm-0.out‚28041.0673828125* proc.memory.percent‚17.578887939453125 disk.rbd1.in‚0 disk.sda3.in‚ 0.47265625$ disk./.usageGB‚582.0167121887207 disk.sda1.out‚0 disk.sda1.in‚0€Ê’õ ‚(:‡ ¸ëÒ¤–®È gpu.0.gpu‚82.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚343.33' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é#âö ‚(:× ¸ëÒÈ®ÃÉ disk.sda1.in‚0! proc.memory.rssMB‚ 2880.765625 disk.sda1.out‚0 disk.sda3.out‚ 172.484375 disk.rbd0.in‚0 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0 proc.cpu.threads‚17) proc.memory.availableMB‚ 1307.30078125$ disk./.usageGB‚582.0167808532715 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 172.484375 disk.sda.out‚ 172.484375( disk.nvme0c0n1.out‚28049.83447265625# disk.dm-0.out‚28049.83447265625$ memory_percent‚92.02086925506592 network.recv‚ 1.126613e+06 disk.rbd1.in‚0 disk.drbd2165.in‚0) proc.memory.percent‚17.58279800415039& disk.nvme0n1.out‚28049.83447265625 cpu‚25.5174901507107 disk.sda2.out‚0 network.sent‚ 4.783762e+06 disk.sda3.in‚ 0.47265625 disk.rbd0.out‚0 disk.rbd1.out‚ 11.828125 disk.dm-0.in‚ 5382.5390625) disk./.usagePercent‚39.05963450766925 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 5382.5390625 disk.nvme0n1.in‚ 5382.5390625ž3'¸ã÷ ‚(:Ø ÇëÒŽÈ¹Ï disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 172.6328125! proc.memory.rssMB‚ 2880.765625) disk./.usagePercent‚39.05963220360144' disk.nvme0c0n1.out‚28060.2705078125" disk.dm-0.out‚28060.2705078125 disk.sda3.out‚ 172.6328125 disk.sda.out‚ 172.6328125' proc.memory.availableMB‚ 1307.546875$ disk./.usageGB‚582.0167465209961 disk.drbd2165.out‚0 disk.sda3.in‚ 0.47265625 proc.cpu.threads‚17 disk.sda2.in‚0 cpu‚25.51825628063481 network.sent‚ 4.790667e+06 network.recv‚ 1.130063e+06 disk.drbd2165.in‚0 disk.dm-0.in‚ 5383.37109375% disk.nvme0n1.out‚28060.2705078125 disk.sda2.out‚0 disk.rbd1.in‚0! disk.nvme0n1.in‚ 5383.37109375 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚11.96875 disk.sda.in‚ 0.47265625$ memory_percent‚92.01936721801758) proc.memory.percent‚17.58279800415039# disk.nvme0c0n1.in‚ 5383.37109375 disk.sda1.out‚0 disk.rbd0.out‚0í? Z“ø ‚(:ˆ ÇëÒºÊÝÏ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚164.055' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.811 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½‰7jŸù ‚(:” ÖëÒ„¯·Ç gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚269.838' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.967600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿p¢ãú ‚(:Ø ÖëÒÁÇÐÉ disk.nvme0n1.in‚ 5442.671875 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚28063.35302734375 disk.dm-0.in‚ 5442.671875' proc.memory.availableMB‚ 1307.296875 disk.sda3.out‚ 172.90234375 disk.rbd0.in‚0 disk.rbd1.out‚11.96875! disk.nvme0c0n1.in‚ 5442.671875$ memory_percent‚92.02089309692383! proc.memory.rssMB‚ 2880.765625 network.recv‚ 1.133239e+06 disk.sda3.in‚ 0.47265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0# disk.dm-0.out‚28063.35302734375& disk.nvme0n1.out‚28063.35302734375 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 172.90234375) proc.memory.percent‚17.58279800415039 proc.cpu.threads‚17 network.sent‚ 4.793631e+06$ disk./.usageGB‚582.0156364440918 disk.rbd1.in‚0 disk.sda.out‚ 172.90234375 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.51865522459283) disk./.usagePercent‚39.05955770540898cŸ69”û ‚(:‰ åëÒ”õÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚113.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈzÄëü ‚(:à åëÒ¾ÕéÉ disk.sda.out‚ 173.44921875 disk.dm-0.in‚ 5444.1171875 disk.nvme0n1.in‚ 5444.1171875) proc.memory.availableMB‚ 1308.05078125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 173.44921875* proc.memory.percent‚17.578959465026855 disk.dm-1.in‚ 0.47265625 disk.drbd2165.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚28072.83642578125# disk.dm-0.out‚28072.83642578125$ memory_percent‚92.01629161834717# proc.memory.rssMB‚ 2880.13671875 network.recv‚ 1.136774e+06 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚12.03125 disk.sda.in‚ 0.47265625& disk.nvme0n1.out‚28072.83642578125 disk.sda1.out‚0) disk./.usagePercent‚39.05959098638843" disk.nvme0c0n1.in‚ 5444.1171875 disk.sda1.in‚0 disk.dm-1.out‚ 173.44921875 network.sent‚ 4.798597e+06$ disk./.usageGB‚582.0161323547363 disk.sda3.in‚ 0.47265625 cpu‚25.51942486281099 disk.sda2.in‚0ýˆ(2ý - _wandbruntime‚4475  _runtime‚4475`I64Æþ À $ _timestamp‚1.784334314607697e+09 _runtime‚4475.192792218 _step‚206 & lr-AdamW/pg1‚0.0004683032104323163 & lr-AdamW/pg2‚0.0004683032104323163 trainer/global_step‚2549¥P½òËÿ Å _step‚206 & lr-AdamW/pg1‚0.0004683032104323163 & lr-AdamW/pg2‚0.0004683032104323163 trainer/global_step‚2549 $ _timestamp‚1.784334314607697e+09 _runtime‚4475.192792218ÎêPkP2€- _wandbruntime‚4475  _runtime‚4475d¡^Tï'éO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚198070764425216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43785245 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22063544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3093749873152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚731957226463232 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21721707 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43795224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2393706936964 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21721701 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _runtime‚4475.194135734 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚734355061288448 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21721701 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚734353689725440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚731957226463232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7242308 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2396463262208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201164919023104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198071132516352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3093786506752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚734034208713348 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22073517 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6542362 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7242578 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚731957226463232 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21721707 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43795224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2393706936964 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21721701 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _runtime‚4475.194135734 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚734355061288448 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21721701 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚734353689725440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚731957226463232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7242308 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2396463262208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201164919023104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198071132516352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3093786506752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚734034208713348 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22073517 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6542362 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7242578 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚734632615491072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚732235227193344 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6544905 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13790240 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13789975 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚732235253366784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚2549 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6544900 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201243181732352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 % _timestamp‚1.7843343160357454e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21729907 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚734633987061248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3094967525888 _runtime‚4476.621162955 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚732235227193344 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22082009 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21729907 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22072022 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3094930899456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2398733694464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43811922 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43801929 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2398733694464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22082009 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198148618924032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2394630403322 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43811922 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201243586449920 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22072022 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192q¬r³þ&…øM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚734313013957882 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2397388297728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚734311643929480 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13789975 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚732235253366784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚2549 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6544900 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201243181732352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 % _timestamp‚1.7843343160357454e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21729907 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚734633987061248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3094967525888 _runtime‚4476.621162955 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚732235227193344 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22082009 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21729907 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22072022 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3094930899456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2398733694464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43811922 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43801929 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2398733694464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22082009 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198148618924032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2394630403322 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43811922 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201243586449920 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22072022 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21729913 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚734632615491072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚732235227193344 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6544905 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13790240 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7381636 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205098458452992 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚747973164706966 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚745534565257216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22489296 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44631276 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6669512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚748300042161664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2442779192832 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚205098861080576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3153408180224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚747974534735256 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14051148 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22132001 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2444124582400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44621297 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3153373643776 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚745857262968832 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22132007 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22489296 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚748301413724672 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22499269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2444124582400 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6669507 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0¶÷Î ó'˜íO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14050888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚211 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22132001 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22499269 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7381636 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205098458452992 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚747973164706966 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚745534565257216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22489296 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44631276 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6669512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚748300042161664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2442779192832 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚205098861080576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3153408180224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚747974534735256 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14051148 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22132001 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2444124582400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44621297 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3153373643776 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚745857262968832 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22132007 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22489296 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚748301413724672 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22499269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2444124582400 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6669507 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚745857289142272 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7381381 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚748301413724672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚748300042161664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201945452900352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚2599 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 $ _timestamp‚1.784334399182047e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚745857289142272 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22132007Ó­nxû™‚(:’ ¿ëÒ‚€”È gpu.0.gpu‚27.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚329.225' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.84500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àWÜÚØš‚(:Í ¿ëÒ àË disk.sda3.out‚ 174.859375 disk.sda.in‚ 0.47265625$ memory_percent‚92.01221466064453 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚12.25 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda.out‚ 174.859375! proc.memory.rssMB‚ 2880.140625 disk.sda2.out‚0 network.recv‚ 1.157901e+06# disk.nvme0n1.out‚28118.41015625 disk.drbd2165.in‚0 disk.dm-0.out‚28118.41015625* proc.memory.percent‚17.578983306884766 network.sent‚ 4.899949e+06) disk./.usagePercent‚39.05969338940212 cpu‚25.523554078426777# disk.nvme0c0n1.in‚ 5519.47265625& proc.memory.availableMB‚ 1308.71875! disk.nvme0n1.in‚ 5519.47265625 disk.dm-1.out‚ 174.859375 disk.drbd2165.out‚0$ disk./.usageGB‚582.0176582336426 disk.dm-1.in‚ 0.47265625% disk.nvme0c0n1.out‚28118.41015625 disk.dm-0.in‚ 5519.47265625 disk.sda1.out‚0 disk.sda2.in‚0•ú2›- _wandbruntime‚4561  _runtime‚4561^Òásø&œòM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚748580339497472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚212 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚745812447035392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22507761 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44647974 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2443704228352 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚746135263698944 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14056177 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚4561.196319132 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6672050 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7384377 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22497774 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚746135289872384 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2445049625088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22497774 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚202022939308032 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44647974 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 $ _timestamp‚1.784334400610969e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205177538321408 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22140207 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14056177 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚4561.196319132 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6672050 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7384377 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22497774 [GDeviceStatsMonitor.on_train_batch_end/active_byte|aZ÷s.large_pool.allocated‚746135289872384 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2445049625088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22497774 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚202022939308032 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44647974 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 $ _timestamp‚1.784334400610969e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205177538321408 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22140207 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚748580339497472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚212 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚745812447035392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22507761 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44647974 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2443704228352 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚746135263698944 EmËŸ©‚(:” Š‘ëÒº×ÁÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚295.159' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.031800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yØ–á窂(:Ü Š‘ëÒ’«¼É! proc.memory.rssMB‚ 2880.765625$ disk./.usageGB‚582.0171508789062 disk.rbd0.out‚0$ memory_percent‚92.01738834381104) proc.memory.availableMB‚ 1307.87109375 proc.cpu.threads‚17 disk.sda.out‚ 176.18359375 disk.dm-0.in‚ 5608.11328125 disk.sda1.out‚0 disk.sda3.out‚ 176.18359375) proc.memory.percent‚17.58279800415039 disk.dm-1.out‚ 176.18359375 disk.rbd1.out‚ 12.453125# disk.nvme0c0n1.in‚ 5608.11328125 disk.sda3.in‚ 0.47265625* disk./.usagePercent‚39.059659340400074 disk.sda1.in‚0! disk.nvme0n1.in‚ 5608.11328125$ disk.nvme0n1.out‚28152.482421875 disk.sda2.in‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚28152.482421875! disk.dm-0.out‚28152.482421875 disk.rbd0.in‚0 cpu‚25.52684167757843 disk.sda2.out‚0 disk.sda.in‚ 0.47265625 network.sent‚ 5.005502e+06 network.recv‚ 1.175564e+06 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0 disk.drbd2165.in‚0¼A2«- _wandbruntime‚4643  _runtime‚4643’©3ƬÀ & lr-AdamW/pg1‚0.0004669683934696152 & lr-AdamW/pg2‚0.0004669683934696152 trainer/global_step‚2649 $ _timestamp‚1.784334483340271e+09 _runtime‚4643.925132566 _step‚214/uGË­Å _runtime‚4643.925132566 _step‚214 & lr-AdamW/pg1‚0.0004669683934696152 & lr-AdamW/pg2‚0.0004669683934696152 trainer/global_step‚2649 $ _timestamp‚1.784334483340271e+09Ö‰™ž2®- _wandbruntime‚4643  _runtime‚4643y¥Ô|î'¯èO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22915048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2490440513024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14316284 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7519876 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2486181163180 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _runtime‚4643.926397414 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚759757325647872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7519627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205819773284352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚2649 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45457349 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22925021 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚759757325647872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2489095123456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚762247766160896 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205819405193216 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3213194492928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2489095123456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14316538 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚762246394597888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22542307 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2490440513024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22542307 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2484836562858 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚762246394597888 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3213229029376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45467328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _step‚215 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6796662 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22925021 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209032599686144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚761914860757164 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22915048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2490440513024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14316284 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7519876 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2486181163180 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _runtime‚4643.926397414 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚759757325647872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7519627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205819773284352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚2649 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45457349 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22925021 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚759757325647872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2489095123456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚762247766160896 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205819405193216 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3213194492928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2489095123456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14316538 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚762246394597888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22542307 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2490440513024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22542307 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2484836562858 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚762246394597888 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3213229029376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648×ñÑeÐ2±- _wandbruntime‚4645  _runtime‚46456|üÆù&²óM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22550507 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚762526691933696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3214394491904 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2487104629538 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚760035300204544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14321780 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14321538 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚762192295973296 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2485760029104 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚759706535944192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2649 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2490020158976 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _step‚216 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚760035300204544 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7522338 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 % _timestamp‚1.7843344847421355e+09 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22933513 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2491365555712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209111286092800 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205896891600896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6799200 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45484026 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45474033 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚760035326377984 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45474033 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22933513 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7522575 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3214429021184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22550513 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2491365555712 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45484026 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚762193666001698 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3214394491904 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2487104629538 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚760035300204544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14321780 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14321538 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚762192295973296 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2485760029104 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚759706535944192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2649 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2490020158976 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _step‚216 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚760035300204544 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7522338 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 % _timestamp‚1.7843344847421355e+09 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22933513 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2491365555712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209111286092800 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205896891600896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6799200 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45484026 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45474033 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚760035326377984 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45474033 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22933513 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7522575 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3214429021184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22550513 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2491365555712 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45484026 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚762193666001698 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6923812 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚209693725577216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3272984624128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚776192747034112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22952607 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7657796 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚773657335980032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23340800 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2535411054080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22952601 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚2699 _step‚219 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23340800 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22952601 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3273019160576 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚775855186779072 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚773657335980032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚776194118597120 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚776194118597120 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 % _timestamp‚1.7843345677900746e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2531073675966 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212966710201344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚775853816750782 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46303380 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚773322768502784 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2536756443648 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968Gˆ%ô'ÈîO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚776194118597120 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚776194118597120 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 % _timestamp‚1.7843345677900746e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2531073675966 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212966710201344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚775853816750782 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46303380 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚773322768502784 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2536756443648 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6923812 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚209693725577216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3272984624128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚776192747034112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22952607 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7657796 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚773657335980032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23340800 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2535411054080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22952601 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚2699 _step‚219 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23340800 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22952601 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3273019160576 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚775855186779072 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚773657335980032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ÛixR`2É- _wandbruntime‚4729  _runtime‚4729z™Œø&ÊòM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784334569196679e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22960807 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209771211984896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2537681486336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22960807 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚2699 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚776473044369920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23359265 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6926355 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚213045795787776 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7660778 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚776473044369920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14587133 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23349278 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚773935362883584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46320078 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3274181182464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚773600650280960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22960813 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2536336089600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚4729.782055216 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14586882 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3274215711744 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚773935336710144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2536336089600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46320078 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995ä¸,òöË÷M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22960813 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚213045393167360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚776473044369920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23359265 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6926355 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚213045795787776 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7660778 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚776473044369920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14587133 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23349278 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚773935362883584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46320078 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3274181182464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚773600650280960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22960813 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2536336089600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚4729.782055216 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14586882 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3274215711744 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚773935336710144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2536336089600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46320078 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784334569196679e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22960807 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209771211984896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2537681486336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22960807 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚2699 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚790139099470336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚787557398659072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚787216857411584 % _timestamp‚1.7843346522905815e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23362907 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚787557372485632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23766552 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2581726984704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚216900396195840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3332350234624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47139432 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7796139 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚213568045961216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚787216831983616 _step‚223 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14847096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚213568414052352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚789795512800980 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2581726984704 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7050962 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚790139099470336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2583072374272 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚790139099470336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚787557398659072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚787216857411584 % _timestamp‚1.7843346522905815e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23362907 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚787557372485632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23766552 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2581726984704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚216900396195840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3332350234624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47139432 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7796139 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚213568045961216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚787216831983616 _step‚223 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14847096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚213568414052352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚789795512800980 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2581726984704 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7050962 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/acti­ç9¢/ve.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚790139099470336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2583072374272 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 Iù&àóM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚2749 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2582652020224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7799156 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚213645900460032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7053505 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3333566581760 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚787835399389184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14852661 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47146137 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7798898 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23371107 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23371113 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚790419396806144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚216979062324224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚213645532368896 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚787835399389184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47156130 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23371107 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23775030 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚787494739189760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚790418025235968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23371113 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23785017 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23785017 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 % _timestamp‚1.7843346537398384e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2582652020224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚787494713761792 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14852398 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚787835399389184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14852661 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47146137 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7798898 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23371107 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23371113 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚790419396806144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚216979062324224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚213645532368896 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚787835399389184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47156130 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23371107 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23775030 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚787494739189760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚790418025235968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23371113 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23785017 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23785017 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 % _timestamp‚1.7843346537398384e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2582652020224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚787494713761792 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14852398 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 % _timestamp‚1.7843347365155065e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚804086823469568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚801457408991232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2629388304896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2629388304896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚804086823469568 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚804085451906560 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚803735838822888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚227 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2628042915328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23773201 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15112776 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3392029832704 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47975484 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚803734468794598 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220834798805504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2623547902182 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚801457408991232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚217442734436352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7178112 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚2799 _runtime‚4897.103009211 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚801457435164672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23773207 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24202277 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097!êHLô'öîO WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7934664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3392064369152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚801457408991232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚217442734436352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7178112 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚2799 _runtime‚4897.103009211 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚801457435164672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23773207 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24202277 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23773201 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47965505 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2628042915328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚220834396177920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23773207 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7934416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2624892502504 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24192304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚801110920892416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚804085451906560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚801110946320384 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24192304 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚217442366345216 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 % _timestamp‚1.7843347365155065e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚804086823469568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚801457408991232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2629388304896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2629388304896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚804086823469568 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚804085451906560 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚803735838822888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚227 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2628042915328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23773201 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15112776 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3392029832704 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47975484 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚803734468794598 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220834798805504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2623547902182 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15117812 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7180655 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7937162 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47992182 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚804365749242368 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚804365749242368 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2630313347584 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217519852752896 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚228 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4898.546461666 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23781413 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚801735409721344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2630313347584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7937397 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2624471368428 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15118052 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚804013274039020 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚2799 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2625815968862 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47982189 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47982189 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚801388828098560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚801735409721344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47992182 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚801735435894784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚220913481321984 % _timestamp‚1.7843347379609177e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24210769 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24200782 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2628967950848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0@CdÉþ&ùøM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23781407 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24200782 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2628967950848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚804364377672192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23781413 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15117812 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7180655 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7937162 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47992182 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚804365749242368 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚804365749242368 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2630313347584 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217519852752896 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚228 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4898.546461666 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23781413 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚801735409721344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2630313347584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7937397 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2624471368428 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15118052 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚804013274039020 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚2799 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2625815968862 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47982189 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47982189 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚801388828098560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚801735409721344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47992182 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚801735435894784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚220913481321984 % _timestamp‚1.7843347379609177e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24210769 P&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04656143859028816 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3063638210296631 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7321513891220093 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10563069581985474 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6616926193237305 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9906229376792908 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.36885926127433777 particle/accuracy_1_of_2‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 epoch‚6 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12527374923229218 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 jet/accuracy_2_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 *jet/accuracy_1_of_4‚0.9630591869354248 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6944444179534912 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7713049054145813 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6671650409698486 jet/accuracy_5_of_5‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.29629626870155334 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8233559727668762 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6978039741516113 /particle/accuracy_4_of_4‚0.9372095465660095 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5988501906394958 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.3150126338005066 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27973073720932007 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2704576253890991 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3000095784664154 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5632981061935425 +jet/accuracy_3_of_3‚0.06840924173593521 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3076923191547394 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9942914247512817 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,jet/accuracy_4_of_4‚0.011098379269242287 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2694353461265564 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5406978726387024 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8240740299224854 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4097222089767456 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.995812177658081 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2034708857536316 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5678960084915161 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975579380989075 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2682364881038666 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.27086976170539856 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.3053285479545593 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6029089689254761 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8192742466926575 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 )jet/accuracy_1_of_2‚0.761651873588562 *jet/accuracy_1_of_3‚0.9234243035316467 'particle/accuracy‚0.849249005317688 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3076923191547394 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.604084312915802 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5127251744270325 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2120181918144226 *particle/sensitivity‚0.626357913017273 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.29336753487586975 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2808096408843994 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_5‚1 particle/accuracy_1_of_3‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9959026575088501 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04089854285120964 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6944444179534912 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5469425320625305 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7344512939453125 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9873397350311279 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9854875206947327 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.033820588141679764 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 6validation_average_jet_accuracy‚0.3457273244857788 _step‚230 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6572890877723694 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9897507429122925 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6041666269302368 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6346028447151184 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.239342600107193 _runtime‚4927.027553127 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.707703709602356 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9959026575088501 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2705538868904114 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 +jet/accuracy_2_of_2‚0.14224347472190857 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6123414039611816 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6117675304412842 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7713049054145813 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8461538553237915 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.29629626870155334 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7052720785140991 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16485583782196045 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9102044105529785 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2819936275482178 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_1‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5743590593338013 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07684379070997238 +jet/accuracy_1_of_1‚0.29909929633140564 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8579033613204956 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2561081349849701 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5285719037055969 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16789491474628448 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.28026384115219116 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.533750057220459 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_1_of_5‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.044321127235889435 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6929357647895813 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2578319013118744 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 )loss/val_total_loss‚9.213721736012749 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44093573093414307 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3177083134651184 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.49259430170059204 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9863324761390686 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2789621353149414 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26232486963272095 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2974476218223572 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 jet/accuracy_3_of_5‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9930908679962158 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5966100096702576 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45756402611732483 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9915128350257874 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9795013666152954 particle/accuracy_3_of_3‚1 +jet/accuracy_3_of_4‚0.08631718903779984bÚÏsOƒ·{ =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9915128350257874 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9795013666152954 particle/accuracy_3_of_3‚1 +jet/accuracy_3_of_4‚0.08631718903779984 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8856799006462097 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5612985491752625 'particle/f_score‚0.6408487558364868 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9969778656959534 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4664333760738373 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9825825691223145 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9922727942466736 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.36775344610214233 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6596121788024902 particle/accuracy_3_of_4‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2705538868904114 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5220215320587158 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.769727885723114 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9968975782394409 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.3070087432861328 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2719702124595642 @&Purity/2FRt1SRqqt2FBt/event_proportion‚»çÐXn72.325148852833081e-05 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8169311285018921 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4804932177066803 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13651864230632782 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32625049352645874 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9221175909042358 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.49932828545570374 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 trainer/global_step‚2799 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3781924545764923 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9934055805206299 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9179011583328247 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9818987846374512 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5447076559066772 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1274999976158142 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5564936995506287 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.643214225769043 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9957697987556458 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4791666567325592 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10563069581985474 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5531473755836487 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8068405389785767 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5521400570869446 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5934000015258789 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6606621742248535 *jet/accuracy_2_of_4‚0.7757135033607483 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1257089078426361 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5658769607543945 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9953507781028748 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45012229681015015 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08769441395998001 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.3607015013694763 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.137654110789299 particle/accuracy_2_of_2‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.28142738342285156 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 % _timestamp‚1.7843347664411623e+09 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8629015684127808 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.49259430170059204 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4951080083847046 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5547507405281067 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5170252919197083 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04656143859028816 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3063638210296631 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7321513891220093 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10563069581985474 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6616926193237305 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9906229376792908 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.36885926127433777 particle/accuracy_1_of_2‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 epoch‚6 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12527374923229218 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 jet/accuracy_2_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 *jet/accuracy_1_of_4‚0.9630591869354248 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6944444179534912 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7713049054145813 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6671650409698486 jet/accuracy_5_of_5‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.29629626870155334 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8233559727668762 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6978039741516113 /particle/accuracy_4_of_4‚0.9372095465660095 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5988501906394958 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.3150126338005066 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27973073720932007 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2704576253890991 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3000095784664154 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5632981061935425 +jet/accuracy_3_of_3‚0.06840924173593521 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3076923191547394 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9942914247512817 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,jet/accuracy_4_of_4‚0.011098379269242287 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2694353461265564 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5406978726387024 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8240740299224854 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4097222089767456 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.995812177658081 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2034708857536316 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5678960084915161 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975579380989075 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2682364881038666 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.27086976170539856 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.3053285479545593 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6029089689254761 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8192742466926575 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 )jet/accuracy_1_of_2‚0.761651873588562 *jet/accuracy_1_of_3‚0.9234243035316467 'particle/accuracy‚0.849249005317688 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3076923191547394 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.604084312915802 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5127251744270325 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2120181918144226 *particle/sensitivity‚0.626357913017273 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.29336753487586975 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2808096408843994 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_5‚1 particle/accuracy_1_of_3‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9959026575088501 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04089854285120964 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6944444179534912 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5469425320625305 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7344512939453125 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9873397350311279 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9854875206947327 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.033820588141679764 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 6validation_average_jet_accuracy‚0.3457273244857788 _step‚230 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6572890877723694 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9897507429122925 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6041666269302368 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6346028447151184 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.239342600107193 _runtime‚4927.027553127 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.707703709602356 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9959026575088501 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2705538868904114 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 +jet/accuracy_2_of_2‚0.14224347472190857 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6123414039611816 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6117675304412842 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7713049054145813 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8461538553237915 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.29629626870155334 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7052720785140991 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16485583782196045 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9102044105529785 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2819936275482178 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_1‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5743590593338013 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07684379070997238 +jet/accuracy_1_of_1‚0.29909929633140564 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8579033613204956 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2561081349849701 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5285719037055969 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16789491474628448 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.28026384115219116 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.533750057220459 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_1_of_5‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.044321127235889435 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6929357647895813 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2578319013118744 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 )loss/val_total_loss‚9.213721736012749 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44093573093414307 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3177083134651184 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.49259430170059204 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9863324761390686 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2789621353149414 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26232486963272095 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2974476218223572 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 jet/accuracy_3_of_5‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9930908679962158 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5966100096702576 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45756402611732483æi5Zž„‚(:“ ¶“ëÒ«¥üÇ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚330.913' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.18260000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡vínä…‚(:Ù ¶“ëÒ“íÈÉ) disk./.usagePercent‚39.27417342949823 disk.drbd2165.in‚0 disk.rbd1.out‚13.21875 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 6015.453125 disk.dm-0.in‚ 6015.453125 disk.nvme0n1.in‚ 6015.453125 disk.sda3.out‚ 184.41796875 proc.cpu.threads‚17# disk./.usageGB‚585.213565826416 disk.rbd1.in‚0( disk.nvme0c0n1.out‚32426.66259765625 disk.sda1.out‚0& disk.nvme0n1.out‚32426.66259765625 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 184.41796875! proc.memory.rssMB‚ 2898.734375 disk.sda.out‚ 184.41796875 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.sda2.in‚0) proc.memory.percent‚17.69247055053711 network.sent‚ 5.337653e+06$ memory_percent‚88.12739849090576) proc.memory.availableMB‚ 1945.20703125 disk.sda.in‚ 0.47265625 network.recv‚ 1.249864e+06# disk.dm-0.out‚32426.66259765625 disk.rbd0.in‚0 disk.sda2.out‚0 cpu‚25.54537765141109œ‰¸·”†‚(:‰ Å“ëÒÿøÇ gpu.0.gpu‚97.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚336.753' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.3506 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥m æ‡‚(:Û Å“ëÒÑƳÉ disk.rbd0.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚33629.86376953125$ memory_percent‚88.13138008117676$ disk./.usageGB‚585.1223449707031 disk.sda.out‚ 184.69140625 disk.drbd2165.out‚0 proc.cpu.threads‚17( proc.memory.availableMB‚ 1944.5546875 network.sent‚ 5.395277e+06) disk./.usagePercent‚39.26805152133194 disk.sda3.out‚ 184.69140625 disk.dm-1.out‚ 184.69140625 disk.rbd1.out‚13.28125 cpu‚25.54484067971337 disk.sda.in‚ 0.47265625 disk.sda2.out‚0 network.recv‚ 1.254594e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0 proc.memory.rssMB‚2899.25) proc.memory.percent‚17.69561767578125( disk.nvme0c0n1.out‚33629.86376953125 disk.dm-0.in‚ 6089.47265625 disk.rbd0.in‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 6089.47265625! disk.nvme0n1.in‚ 6089.47265625# disk.dm-0.out‚33629.86376953125£Ú䔈‚(:‰ Ô“ëÒ ùÌÇ gpu.0.gpu‚72.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚335.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.1284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü”׉‚(:Ì Ô“ëÒ¼ñËÉ disk.drbd2165.in‚0 cpu‚25.54435314639554 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd1.out‚13.28125 disk.sda.out‚ 185.0546875 disk.sda3.in‚ 0.47265625 disk.sda1.in‚0 disk.sda3.out‚ 185.0546875 disk.rbd1.in‚0$ memory_percent‚88.13285827636719% proc.memory.availableMB‚ 1944.3125 proc.memory.rssMB‚2899.25) proc.memory.percent‚17.69561767578125 network.sent‚ 5.400251e+06$ disk.nvme0n1.out‚33637.517578125 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 6090.3046875 disk.dm-1.out‚ 185.0546875 disk.sda.in‚ 0.47265625) disk./.usagePercent‚39.32006585210121" disk.nvme0c0n1.in‚ 6090.3046875 disk.dm-0.in‚ 6090.3046875! disk.dm-0.out‚33637.517578125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625 network.recv‚ 1.257849e+06$ disk./.usageGB‚585.8973960876465& disk.nvme0c0n1.out‚33637.517578125 disk.rbd0.out‚0E¢=“Š‚(:ˆ ã“ëÒ³¢ÀÇ gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚282.495' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.499 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°€Åãâ‹‚(:× ã“ëÒÄ«ŠÉ& disk.nvme0n1.out‚34456.62646484375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625! disk.nvme0c0n1.in‚ 6090.828125 cpu‚25.543826128049226 network.recv‚ 1.260597e+06 disk.dm-1.out‚ 185.29296875) disk./.usagePercent‚39.31777612071497$ disk./.usageGB‚585.8632774353027 disk.nvme0n1.in‚ 6090.828125 disk.sda.out‚ 185.29296875 disk.drbd2165.in‚0 disk.dm-0.in‚ 6090.828125( proc.memory.availableMB‚ 1944.5546875 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 disk.dm-1.in‚ 0.47265625) proc.memory.percent‚17.69561767578125# disk.dm-0.out‚34456.62646484375 proc.cpu.threads‚17 network.sent‚ 5.404303e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 13.359375 disk.drbd2165.out‚0 proc.memory.rssMB‚2899.25$ memory_percent‚88.13138008117676 disk.sda3.out‚ 185.29296875( disk.nvme0c0n1.out‚34456.62646484375ŠX‘¤”Œ‚(:‰ ò“ëÒ¿—úÇ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚335.011' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.0022 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Jtç›æ‚(:Û ò“ëÒó‘Ê disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 185.55078125 disk.sda.out‚ 185.55078125# disk.dm-0.out‚34464.25341796875! disk.nvme0n1.in‚ 6091.26953125 disk.sda1.in‚0( disk.nvme0c0n1.out‚34464.25341796875 disk.sda1.out‚0 disk.rbd1.out‚ 13.359375 disk.drbd2165.in‚0$ memory_percent‚88.13142776489258 network.sent‚ 5.409298e+06 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 6091.26953125& disk.nvme0n1.out‚34464.25341796875) proc.memory.percent‚17.69561767578125 disk.sda3.in‚ 0.47265625 network.recv‚ 1.264077e+06$ disk./.usageGB‚585.8633003234863 disk.sda3.out‚ 185.55078125 disk.rbd0.in‚0' proc.memory.availableMB‚ 1944.546875) disk./.usagePercent‚39.31777765676017# disk.nvme0c0n1.in‚ 6091.26953125 cpu‚25.54323575994505 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625 proc.memory.rssMB‚2899.25 proc.cpu.threads‚17¦þ¾#”Ž‚(:‰ ”ëÒ¬ØßÇ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚336.753' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.3506 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-öb>Í‚(: ”ëÒ£û‹É& disk.nvme0n1.out‚34475.88037109375 disk.sda.in‚ 0.47265625 cpu‚25.542720844940387 disk.sda2.out‚0 network.recv‚ 1.267586e+06 disk.rbd0.in‚0$ disk./.usageGB‚585.1225242614746 disk.drbd2165.in‚0# disk.dm-0.out‚34475.88037109375) proc.memory.percent‚17.69561767578125 network.sent‚ 5.414364e+06 disk.nvme0n1.in‚6162 disk.sda3.in‚ 0.47265625 disk.dm-1.out‚ 185.87890625 disk.rbd1.out‚ 13.4296875' proc.memory.availableMB‚ 1944.515625 proc.cpu.threads‚17 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0 disk.nvme0c0n1.in‚6162 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-0.in‚6162$ memory_percent‚88.13161849975586 proc.memory.rssMB‚2899.25) disk./.usagePercent‚39.26806355368605 disk.sda1.in‚0 disk.sda3.out‚ 185.87890625 disk.sda.out‚ 185.87890625( disk.nvme0c0n1.out‚34475.88037109375 disk.sda1.out‚0 disk.drbd2165.out‚0¶?2- _wandbruntime‚5014  _runtime‚5014èðìÇ‘Á & lr-AdamW/pg2‚0.0004641251402624978 trainer/global_step‚2849 % _timestamp‚1.7843348542958817e+09 _runtime‚5014.881240152 _step‚231 & lr-AdamW/pg1‚0.0004641251402624978tiÝÌ’Æ & lr-AdamW/pg2‚0.0004641251402624978 trainer/global_step‚2849 % _timestamp‚1.7843348542958817e+09 _runtime‚5014.881240152 _step‚231 & lr-AdamW/pg1‚0.0004641251402624978çÂœu2“- _wandbruntime‚5014  _runtime‚5014T3«f%”éO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚819185453105152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24315507 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24677043 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚821506532880556 _runtime‚5014.883462838 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚821866999954944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49002529 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚821866999954944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚819185453105152 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7314180 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚821507902908846 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8106199 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7314175 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24315513 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8106446 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7843348542977192e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15420374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224824025559552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24687016 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3459259022848 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2675614146732 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224824426089984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15420626 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24677043 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚821865628391936 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24315507 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2681546849792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24315513 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚821865628391936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2681546849792 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚819185426931712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚819185453105152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24315507 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24677043 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚821506532880556 _runtime‚5014.883462838 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚821866999954944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49002529 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚821866999954944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚819185453105152 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7314180 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚821507902908846 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8106199 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7314175 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24315513 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8106446 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7843348542977192e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15420374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224824025559552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24687016 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3459259022848 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2675614146732 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224824426089984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15420626 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24677043 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚821865628391936 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24315507 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2681546849792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24315513 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚821865628391936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2681546849792 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚819185426931712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843348557226841e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2682471892480 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24685521 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24323713 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3460423076352 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49009234 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15425927 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2676537612978 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8109204 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚5016.308128047 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24695508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚819463453835264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚221442285383680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2681126495744 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚819108825939968 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24695508 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚822145925727744 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3460455508480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚221442653474816 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49019227 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚233 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24323719 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚2849 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚822144554157568 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15425682 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8108964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚821786708153380 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚822145925727744 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49019227 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7316718 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224902708460032ÎÏÈþ&˜øM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚821785338124978 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚819463453835264 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2682471892480 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚819463427661824 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚819463427661824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843348557226841e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2682471892480 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24685521 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24323713 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3460423076352 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49009234 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15425927 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2676537612978 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8109204 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚5016.308128047 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24695508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚819463453835264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚221442285383680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2681126495744 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚819108825939968 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24695508 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚822145925727744 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3460455508480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚221442653474816 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49019227 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚233 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24323719 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚2849 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚822144554157568 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15425682 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8108964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚821786708153380 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚822145925727744 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49019227 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7316718 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224902708460032é*z52™- _wandbruntime‚5016  _runtime‚5016·µjF š„ epoch‚7 % _timestamp‚1.7843348557853646e+09 2loss/SRqqt1/assignment_loss‚0.5653825176746836 1loss/SRqqt1/detection_loss‚0.7848398456267244 %loss/total_loss‚8.754111630121184 trainer/global_step‚2849 0loss/FRt2/assignment_loss‚0.8263817960250565 _step‚234 2loss/FBt2/assignment_loss‚0.012243587955212873 /loss/FRt1/detection_loss‚0.5316290466190408 /loss/FRt2/detection_loss‚1.5178917454768148 1loss/SRqqt2/detection_loss‚0.6851705345864358 0loss/FBt1/detection_loss‚0.43743447459540363 _runtime‚5016.370432367 0loss/FRt1/assignment_loss‚1.7777497016115864 2loss/SRqqt2/assignment_loss‚0.5476413520443743 2loss/FBt1/assignment_loss‚0.019886106380948336 /loss/FBt2/detection_loss‚1.0478609215249033ºs|«›‰ 2loss/FBt2/assignment_loss‚0.012243587955212873 /loss/FRt1/detection_loss‚0.5316290466190408 /loss/FRt2/detection_loss‚1.5178917454768148 1loss/SRqqt2/detection_loss‚0.6851705345864358 0loss/FBt1/detection_loss‚0.43743447459540363 _runtime‚5016.370432367 0loss/FRt1/assignment_loss‚1.7777497016115864 2loss/SRqqt2/assignment_loss‚0.5476413520443743 2loss/FBt1/assignment_loss‚0.019886106380948336 /loss/FBt2/detection_loss‚1.0478609215249033 epoch‚7 % _timestamp‚1.7843348557853646e+09 2loss/SRqqt1/assignment_loss‚0.5653825176746836 1loss/SRqqt1/detection_loss‚0.7848398456267244 %loss/total_loss‚8.754111630121184 trainer/global_step‚2849 0loss/FRt2/assignment_loss‚0.8263817960250565 _step‚234êÕI|üÐœ‚(:’ ”ëÒ­§˜È gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚331.35'ÀÍÍ gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.27000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥ZÓ‚(:· ”ëÒ˜ÿÉ) proc.memory.availableMB‚ 1944.21484375" disk.nvme0n1.out‚ 34480.1171875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚6162 disk.sda3.out‚ 186.19921875 disk.drbd2165.out‚0 proc.memory.rssMB‚2899.25) proc.memory.percent‚17.69561767578125 disk.dm-1.in‚ 0.47265625 disk.rbd1.out‚ 13.4296875$ memory_percent‚88.13345432281494 proc.cpu.threads‚17 network.recv‚ 1.27105e+06 disk.nvme0n1.in‚6162 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.sda.in‚ 0.47265625$ disk.nvme0c0n1.out‚ 34480.1171875 disk.dm-1.out‚ 186.19921875 disk.sda.out‚ 186.19921875 disk.drbd2165.in‚0 cpu‚25.542237865478807 disk.sda1.in‚0 disk.rbd0.out‚0 network.sent‚ 5.419224e+06) disk./.usagePercent‚39.26626817284842$ disk./.usageGB‚585.0957717895508 disk.rbd0.in‚0 disk.dm-0.in‚6162 disk.dm-0.out‚ 34480.1171875 disk.sda2.out‚0yú{¥’ž‚(:‡ Ÿ”ëÒØ­˜Ç gpu.0.gpu‚89.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚341.58' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.316 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07ÁQ⟂(:× Ÿ”ëÒ ‚µË disk.sda1.in‚0 disk.dm-1.in‚ 0.47265625( proc.memory.availableMB‚ 1944.1171875$ disk.nvme0n1.out‚34488.923828125 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 6162.96484375 disk.dm-0.in‚ 6162.96484375 disk.rbd0.out‚0# memory_percent‚88.1340503692627 proc.memory.rssMB‚2899.25 proc.cpu.threads‚17 disk.dm-1.out‚ 186.51171875 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625& disk.nvme0c0n1.out‚34488.923828125) proc.memory.percent‚17.69561767578125$ disk./.usageGB‚585.0953712463379! disk.nvme0n1.in‚ 6162.96484375 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.sda.out‚ 186.51171875 disk.drbd2165.in‚0 cpu‚25.541703223458327 disk.sda3.out‚ 186.51171875 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0! disk.dm-0.out‚34488.923828125 network.recv‚ 1.275879e+06) disk./.usagePercent‚39.26624129205733 disk.rbd1.out‚ 13.5390625 network.sent‚ 5.482235e+06-í—” ‚(:‰ ®”ëÒö¾Ê gpu.0.gpu‚70.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚111.241' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2482 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞÅÑ¡‚(:Æ ®”ëÒ…«Ë$ disk.nvme0c0n1.out‚ 34503.8359375 disk.sda2.in‚0 disk.sda.in‚ 0.47265625 disk.sda.out‚ 188.4453125! disk.nvme0c0n1.in‚ 6163.796875 disk.dm-0.in‚ 6163.796875 cpu‚25.541186971598695 network.sent‚ 5.48732e+06) disk./.usagePercent‚39.26625972459979 disk.dm-0.out‚ 34503.8359375 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 188.4453125$ memory_percent‚88.13409805297852 proc.memory.rssMB‚2899.25' proc.memory.availableMB‚ 1944.109375# disk./.usageGB‚585.095645904541" disk.nvme0n1.out‚ 34503.8359375 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.rbd1.out‚ 13.5390625 network.recv‚ 1.279116e+06 disk.nvme0n1.in‚ 6163.796875 disk.dm-1.in‚ 0.47265625 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 188.4453125) proc.memory.percent‚17.69561767578125 proc.cpu.threads‚17 disk.sda1.out‚0g ”¢‚(:‰ ½”ëÒ¨˜êÇ gpu.0.gpu‚56.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@²å£‚(:Ú ½”ëÒ´ŒçÉ& disk.nvme0n1.out‚34506.69873046875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0$ disk./.usageGB‚585.0957107543945 disk.rbd1.out‚ 13.6640625 network.recv‚ 1.282501e+06 disk.sda2.out‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚34506.69873046875# disk.dm-0.out‚34506.69873046875( proc.memory.availableMB‚ 1944.0859375 cpu‚25.540542016556696 disk.nvme0n1.in‚ 6243.8046875 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 189.19921875 disk.rbd1.in‚0 disk.sda.out‚ 189.19921875 disk.dm-0.in‚ 6243.8046875 disk.sda3.out‚ 189.19921875" disk.nvme0c0n1.in‚ 6243.8046875$ memory_percent‚88.13424110412598) proc.memory.percent‚17.69561767578125 network.sent‚ 5.49228e+06 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 disk.sda2.in‚0 proc.memory.rssMB‚2899.25 disk.sda.in‚ 0.47265625 proc.cpu.threads‚17) disk./.usagePercent‚39.26626407672787VÂq‡”¤‚(:‰ Ì”ëÒÀ’ÉÇ gpu.0.gpu‚68.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äãìÂÑ¥‚(:Æ Ì”ëÒÜ©ÑÉ disk.drbd2165.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 189.5859375 disk.drbd2165.out‚0 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 6244.76953125" disk.nvme0c0n1.out‚ 34517.84375 disk.dm-0.in‚ 6244.76953125! disk.nvme0n1.in‚ 6244.76953125 disk.sda3.out‚ 189.5859375& proc.memory.availableMB‚ 1944.09375 disk.rbd1.out‚ 13.6640625) proc.memory.percent‚17.69561767578125 network.recv‚ 1.285622e+06( disk./.usagePercent‚39.2662530684039$ disk./.usageGB‚585.0955467224121 disk.dm-0.out‚ 34517.84375 disk.sda1.out‚0$ memory_percent‚88.13419342041016 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 disk.nvme0n1.out‚ 34517.84375 disk.sda3.in‚ 0.47265625 network.sent‚ 5.497335e+06 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.540082379288464 disk.sda.out‚ 189.5859375 proc.memory.rssMB‚2899.25 disk.sda.in‚ 0.47265625&-ï2¦- _wandbruntime‚5098  _runtime‚5098«X}ôȧ _runtime‚5098.494245371 _step‚235 ' lr-AdamW/pg1‚0.00046337842124199096 ' lr-AdamW/pg2‚0.00046337842124199096 trainer/global_step‚2899 $ _timestamp‚1.784334937909166e+09 Ì%Âꬂ _step‚235 ' lr-AdamW/pg1‚0.00046337842124199096 ' lr-AdamW/pg2‚0.00046337842124199096 trainer/global_step‚2899 $ _timestamp‚1.784334937909166e+09 _runtime‚5098.494245371ëí T2©- _wandbruntime‚5098  _runtime‚5098Šc@+î'ªèO BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚225239119360000 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3519058665984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚835811980828160 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25112768 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2727862780416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24725807 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49838581 trainer/global_step‚2899 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15686082 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚835448228930754 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚832725007642624 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25102795 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228758546117120 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚832725033070592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25112768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7441330 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49828602 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25102795 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7441325 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚833085489610752 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚835446858902464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚225239487451136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚5098.495797711 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8244752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24725807 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2723195860162 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚833085463437312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2726517390848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 $ _timestamp‚1.784334937910418e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚833085489610752 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3519024129536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _step‚236 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228758143489536!.ˆÈó'«íO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚833085489610752 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚835446858902464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚225239487451136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚5098.495797711 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8244752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24725807 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2723195860162 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚833085463437312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2726517390848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 $ _timestamp‚1.784334937910418e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚833085489610752 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3519024129536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _step‚236 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228758143489536 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚225239119360000 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3519058665984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚835811980828160 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25112768 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2727862780416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24725807 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49838581 trainer/global_step‚2899 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15686082 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚835448228930754 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚832725007642624 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25102795 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228758546117120 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚832725033070592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25112768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7441330 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49828602 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25102795 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7441325 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979ì?k™2¬- _wandbruntime‚5099  _runtime‚5099 |+¢0­óM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚833363490340864 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7443868 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3520212504064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚833363464167424 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25121260 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228836818271744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24734013 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25121260 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15691327 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚225316605767680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚835727034175288 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚2899 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚833002889420800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49855279 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15691065 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚833002914848768 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2727442426368 GADeviceStatsMonitor.on쨬òÉ_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24734013 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚836092278163968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49855279 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228837220892160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2727442426368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8247197 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚836092278163968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49855279 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228837220892160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2727442426368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8247197 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚833363464167424 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25121260 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228836818271744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24734013 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25121260 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15691327 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚225316605767680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚835727034175288 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚2899 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚833002889420800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49855279 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15691065 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚833002914848768 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2727442426368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24734013 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 EÇ2¾- _wandbruntime‚5181  _runtime‚5181cÉ¿Ã ' lr-AdamW/pg1‚0.00046261743447371413 ' lr-AdamW/pg2‚0.00046261743447371413 trainer/global_step‚2949 % _timestamp‚1.7843350213424256e+09 _runtime‚5181.927511768 _step‚239Oh*ÎÀÈ ' lr-AdamW/pg1‚0.00046261743447371413 ' lr-AdamW/pg2‚0.00046261743447371413 trainer/global_step‚2949 % _timestamp‚1.7843350213424256e+09 _runtime‚5181.927511768 _step‚239ï)ìn2Á- _wandbruntime‚5181  _runtime‚5181HHH¬´ÂéO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚846985526116352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25136113 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚846985499942912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2772833321472 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3578354845696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232691794589696 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 trainer/global_step‚2949 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50664654 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚849758333264384 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25136107 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15951929 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25538520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.KøŒò; on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25136107 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25538520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8383190 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7843350213437226e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2774178711040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _runtime‚5181.932068458 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3578389382144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25528547 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25528547 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8383449 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚849759704827392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 _step‚240 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚849759704827392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 _step‚240 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚849758333264384 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25136107 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15951929 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995ðbú2Ä- _wandbruntime‚5183  _runtime‚5183dyø&ÅòM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25144313 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2769011839194 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25547012 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15956930 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8385912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7571023 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232770475360256 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229190926151680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚847263526846464 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50681338 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8386157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚229191294242816 _step‚241 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2773758356992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚847263500673024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚849667360197196 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232770877980672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚846896978329600 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚850038630600192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2775103753728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 $ _timestamp‚1.784335022771315e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25144319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _runtime‚5183.357007548 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3579549208576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50691331 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚847263500673024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15957180 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2773758356992 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50691331 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7571018 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2770356439628 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚849665990168794 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50681338 trainer/global_step‚2949 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25144313 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25537025 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25144319 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚846897003757568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25547012 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3579583737856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25537025 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175Ry<í„$Æ÷M N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚846897003757568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25547012 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3579583737856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25537025 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25144313 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2769011839194 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25547012 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15956930 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8385912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7571023 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232770475360256 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229190926151680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚847263526846464 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50681338 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8386157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚229191294242816 _step‚241 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2773758356992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚847263500673024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚849667360197196 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232770877980672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚846896978329600 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚850038630600192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2775103753728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 $ _timestamp‚1.784335022771315e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25144319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _runtime‚5183.357007548 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3579549208576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50691331 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚847263500673024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15957180 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2773758356992 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50691331 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7571018 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2770356439628 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚849665990168794 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50681338 trainer/global_step‚2949 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25144313 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25537025 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25144319 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚863704685700608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232988128219136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3637855814656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚860513210888192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8520347 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3637890351104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2815670086378 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2814325486056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25954299 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2819149252096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2820494641664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚863704685700608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8520098 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25546413 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚860885562621952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚860885536448512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51500706 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51510685 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7695630 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚236625615942656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚860513185460224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 $ _timestamp‚1.784335105344025e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25546407 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25546407 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚860885562621952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚863706057263616 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232987760128000 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16215723 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2820494641664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 _runtime‚5265.932055866 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25546413 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16215977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚236626018570240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _step‚244 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚863706057263616 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926ŒG>ó'ÛíO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _step‚244 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚863706057263616 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7695625 trainer/global_step‚2999 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚860885536448512 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚863328880974570 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚863327510946280 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25964272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚863704685700608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232988128219136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3637855814656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚860513210888192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8520347 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3637890351104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2815670086378 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2814325486056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25954299 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2819149252096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2820494641664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚863704685700608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8520098 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25546413 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚860885562621952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚860885536448512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51500706 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51510685 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7695630 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚236625615942656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚860513185460224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 $ _timestamp‚1.784335105344025e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25546407 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25546407 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚860885562621952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚863706057263616 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232987760128000 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16215723 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2820494641664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 _runtime‚5265.932055866 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25546413 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16215977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚236626018570240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175ôίü2Ü- _wandbruntime‚5267  _runtime‚5267Ó®vtâÝóM PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚233065614626816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚860791092666368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25554613 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8522803 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚236704298703872 XIDeviceStatsMonitor.on_train_aõ&s#batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚233065246535680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 % _timestamp‚1.7843351067542582e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51527383 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚863984983036416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25962777 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25554619 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25554619 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25972764 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51527383 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7698168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2815248952302 trainer/global_step‚2999 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51517390 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚863984983036416 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2821419684352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚861163563352064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚863606316190702 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚861163537178624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16220971 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2820074287616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3639086697472 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25972764 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25554613 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8523047 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2816593552736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚861163537178624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7698173 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16221220 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚236704701324288 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚863607686219104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚860791067238400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚5267.339771297 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3639052168192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚861163563352064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2821419684352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51517390 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25962777 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚245 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993|`0ýþ&ÞøM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7698168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2815248952302 trainer/global_step‚2999 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51517390 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚863984983036416 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2821419684352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚861163563352064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚863606316190702 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚861163537178624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16220971 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2820074287616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3639086697472 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25972764 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25554613 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8523047 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2816593552736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚861163537178624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7698173 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16221220 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚236704701324288 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚863607686219104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚860791067238400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚5267.339771297 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3639052168192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚861163563352064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2821419684352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51517390 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25962777 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚245 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚233065614626816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚860791092666368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25554613 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8522803 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚236704298703872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚233065246535680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 % _timestamp‚1.7843351067542582e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51527383 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚863984983036416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25962777 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25554619 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25554619 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25972764 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51527383 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648õe¹6¶2ß- _wandbruntime‚5267  _runtime‚5267|I«àý epoch‚7 _step‚246 .loss/FBt1/detection_loss‚0.510071837305963 %loss/total_loss‚9.202945091333588 trainer/global_step‚2999 _runtime‚5267.414811455 0loss/FRt1/assignment_loss‚1.7895172953379017 0loss/FRt2/assignment_loss‚0.9887384646006641 1loss/FBt1/assignment_loss‚0.04012045713535168 2loss/SRqqt1/assignment_loss‚0.5762869316847125 2loss/SRqqt2/assignment_loss‚0.6261520703482747 1loss/FBt2/assignment_loss‚0.04568904339399648 /loss/FRt2/detection_loss‚1.7123561279525745 /loss/FBt2/detection_loss‚0.8909780482022926 $ _timestamp‚1.784335106829653e+09 /loss/FRt1/detection_loss‚0.5556199494648745 1loss/SRqqt1/detection_loss‚0.7650069074068022 /loss/SRqqt2/detection_loss‚0.70240795850018_{ Öá‚ 2loss/SRqqt1/assignment_loss‚0.5762869316847125 2loss/SRqqt2/assignment_loss‚0.6261520703482747 1loss/FBt2/assignment_loss‚0.04568904339399648 /loss/FRt2/detection_loss‚1.7123561279525745 /loss/FBt2/detection_loss‚0.8909780482022926 $ _timestamp‚1.784335106829653e+09 /loss/FRt1/detection_loss‚0.5556199494648745 1loss/SRqqt1/detection_loss‚0.7650069074068022 /loss/SRqqt2/detection_loss‚0.70240795850018 epoch‚7 _step‚246 .loss/FBt1/detection_loss‚0.510071837305963 %loss/total_loss‚9.202945091333588 trainer/global_step‚2999 _runtime‚5267.414811455 0loss/FRt1/assignment_loss‚1.7895172953379017 0loss/FRt2/assignment_loss‚0.9887384646006641 1loss/FBt1/assignment_loss‚0.04012045713535168ösØf”â‚(:‰ –ëÒšÂ£È gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚292.207' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.4414 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R¼uÅßã‚(:Ô –ëÒ°É disk.dm-1.in‚ 0.47265625 disk.drbd2165.out‚0 disk.sda3.in‚ 0.47265625) proc.memory.availableMB‚ 1944.16015625 network.recv‚ 1.333815e+06 disk.rbd1.out‚ 14.2265625 disk.sda.out‚ 192.95703125# disk.nvme0c0n1.in‚ 6496.67578125! disk.nvme0n1.in‚ 6496.67578125) proc.memory.percent‚17.69561767578125 network.sent‚ 5.678118e+06$ disk./.usageGB‚585.0969657897949 disk.drbd2165.in‚0# disk.nvme0n1.out‚34613.52734375 disk.sda1.in‚0 disk.sda3.out‚ 192.95703125 disk.rbd0.out‚0 disk.sda2.out‚0) disk./.usagePercent‚39.26634830320664% disk.nvme0c0n1.out‚34613.52734375 cpu‚25.5337133282503 proc.cpu.threads‚17 disk.dm-1.out‚ 192.95703125 disk.sda.in‚ 0.47265625 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 6496.67578125 disk.dm-0.out‚34613.52734375 disk.rbd0.in‚0$ memory_percent‚88.13378810882568 disk.sda1.out‚0 proc.memory.rssMB‚2899.25ßÔ-°’ä‚(:‡ ž–ëÒ£ññÇ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚242.96' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.592 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üÓ³¾æå‚(:Û ž–ëÒÿ¼ùÉ proc.cpu.threads‚17& disk.nvme0n1.out‚34624.36669921875 disk.sda3.out‚ 193.10546875 disk.drbd2165.in‚0$ memory_percent‚88.13536167144775 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 193.10546875 disk.rbd1.out‚ 14.2265625 disk.sda.out‚ 193.10546875 cpu‚25.53323408572654$ disk./.usageGB‚585.0971145629883" disk.nvme0c0n1.in‚ 6497.5078125 disk.dm-0.in‚ 6497.5078125 disk.nvme0n1.in‚ 6497.5078125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 proc.memory.rssMB‚2899.25 disk.sda.in‚ 0.47265625) proc.memory.percent‚17.69561767578125 network.recv‚ 1.338399e+06( disk.nvme0c0n1.out‚34624.36669921875 disk.rbd1.in‚0 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1943.90234375) disk./.usagePercent‚39.26635828750047 disk.sda1.in‚0 disk.dm-1.in‚ 0.47265625 disk.sda3.in‚ 0.47265625 network.sent‚ 5.741033e+06# disk.dm-0.out‚34624.36669921875q–V“æ‚(:ˆ ­–ëÒÐÀ©Ç gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚153.445' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.689 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú'Ï,êç‚(:ß ­–ëÒÃûÈ network.sent‚ 5.745992e+06 network.recv‚ 1.342097e+06# disk.dm-0.out‚34633.40869140625! disk.nvme0n1.in‚ 6498.47265625 disk.sda3.in‚ 0.47265625$ disk./.usageGB‚585.0968627929688 disk.sda1.in‚0 disk.rbd1.in‚0$ memory_percent‚88.13397884368896& disk.nvme0n1.out‚34633.40869140625 disk.drbd2165.in‚0 disk.dm-0.in‚ 6498.47265625) proc.memory.percent‚17.69561767578125 cpu‚25.53277563952895 proc.memory.rssMB‚2899.25 disk.sda1.out‚0( disk.nvme0c0n1.out‚34633.40869140625 disk.rbd0.in‚0 disk.sda.out‚ 193.34765625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 disk.rbd1.out‚ 14.2890625) proc.memory.availableMB‚ 1944.12890625# disk.nvme0c0n1.in‚ 6498.47265625 disk.sda2.out‚0 disk.dm-1.out‚ 193.34765625* disk./.usagePercent‚39.266341391003216 disk.drbd2165.out‚0 disk.sda3.out‚ 193.34765625 disk.rbd0.out‚0 disk.sda.in‚ 0.47265625rŠl˜”è‚(:‰ ¼–ëÒÍƘÇ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚232.459' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.4918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CVbæé‚(:Û ¼–ëÒ†·ðÈ network.recv‚ 1.345448e+06 disk.sda2.out‚0 disk.sda.in‚ 0.47265625 disk.sda.out‚ 193.5078125 disk.drbd2165.in‚0 disk.drbd2165.out‚0) proc.memory.percent‚17.69561767578125 proc.cpu.threads‚17) disk./.usagePercent‚39.26633703887513$ disk./.usageGB‚585.0967979431152 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 14.2890625( proc.memory.availableMB‚ 1943.8671875 proc.memory.rssMB‚2899.25 network.sent‚ 5.750998e+06 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 6575.73046875( disk.nvme0c0n1.out‚34639.59130859375# disk.dm-0.out‚34639.59130859375! disk.nvme0n1.in‚ 6575.73046875 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 193.5078125$ memory_percent‚88.13557624816895 cpu‚25.532310792242225 disk.rbd0.out‚0 disk.dm-1.out‚ 193.5078125& disk.nvme0n1.out‚34639.59130859375 disk.dm-0.in‚ 6575.73046875 disk.sda1.in‚0 ©‚p”ê‚(:‰ Ë–ëÒÚÿÉÇ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚213.059' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.6118 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E±òÖë‚(:Ë Ë–ëÒ‡™ÑÉ network.sent‚ 5.755967e+06$ disk./.usageGB‚585.0968399047852 disk.sda3.out‚ 193.74609375 disk.rbd0.in‚0 disk.drbd2165.out‚0' proc.memory.availableMB‚ 1944.109375 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.sda2.in‚0) proc.memory.percent‚17.69561767578125) disk./.usagePercent‚39.26633985495801 disk.sda1.out‚0 proc.memory.rssMB‚2899.25 cpu‚25.531793628560614 proc.cpu.threads‚17 disk.nvme0n1.in‚ 6576.5625 disk.nvme0c0n1.in‚ 6576.5625 disk.dm-0.in‚ 6576.5625$ memory_percent‚88.13409805297852 disk.drbd2165.in‚0 network.recv‚ 1.348667e+06 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0 disk.rbd1.out‚ 14.3671875 disk.rbd0.out‚0 disk.dm-1.out‚ 193.74609375 disk.sda.in‚ 0.47265625! disk.dm-0.out‚34647.677734375$ disk.nvme0n1.out‚34647.677734375 disk.sda.out‚ 193.74609375& disk.nvme0c0n1.out‚34647.677734375¿§F°2ì- _wandbruntime‚5349  _runtime‚5349ÿÆçÈí ' lr-AdamW/pg1‚0.00046105285411413444 ' lr-AdamW/pg2‚0.00046105285411413444 trainer/global_step‚3049 % _timestamp‚1.7843351888719707e+09 _runtime‚ 5349.45777145 _step‚247 ñ<ÍîÇ ' lr-AdamW/pg2‚0.00046105285411413444 trainer/global_step‚3049 % _timestamp‚1.7843351888719707e+09 _runtime‚ 5349.45777145 _step‚247 ' lr-AdamW/pg1‚0.00046105285411413444÷ËÐßú2ï- _wandbruntime‚5349  _runtime‚5349,qƒGÝðéO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚877652409699840 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚236862448603136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3697380349952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26380051 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚877651038136832 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚874785599127552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2861907199486 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7822780 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25956707 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5Žú8ô'ñîO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26380051 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚877651038136832 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚874785599127552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2861907199486 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7822780 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25956707 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚877652409699840 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚236862448603136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3697380349952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚875063573684224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚236939935010816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2862830665844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25964919 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25964919 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3698563481600 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25964913 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚874685181575168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚240638533021696 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52363435 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚877931335472640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26398516 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚875063573684224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚874685156147200 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚877931335472640 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8660175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7825323 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26398516 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2867735614976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52353442 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25964913 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26388529 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16485493 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2861486065410 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52363435 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚5350.918455542 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚240638130401280 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2867735614976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843351903329096e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚877929963902464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚877548012241012 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚877546642212610 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7825318 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚875063599857664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2866390218240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8660425 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚236939566919680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3698598010880 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7825318 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚875063599857664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2866390218240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8660425 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚236939566919680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3698598010880 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚875063573684224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚236939935010816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2862830665844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25964919 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25964919 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3698563481600 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25964913 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚874685181575168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚240638533021696 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52363435 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚877931335472640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.small_‘ƒÆR pool.allocated‚26398516 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚875063573684224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚874685156147200 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚877931335472640 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8660175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7825323 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26398516 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2867735614976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52353442 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25964913 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26388529 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16485493 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2861486065410 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52363435 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚5350.918455542 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚240638130401280 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2867735614976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255ù@Þç2õ- _wandbruntime‚5350  _runtime‚5350QLrö trainer/global_step‚3049 $ _timestamp‚1.784335190378266e+09 _runtime‚5350.963497324 /loss/FRt1/detection_loss‚0.5329966346416594 epoch‚7 _step‚250 /loss/FRt1/assignment_loss‚1.815671792414733 3loss/FBt2/assignment_loss‚0.0006135418826327717 /loss/FBt2/detection_loss‚0.9102748233292784 %loss/total_loss‚8.926061448261404 0loss/FRt2/assignment_loss‚1.0359809959881774 2loss/SRqqt1/assignment_loss‚0.5490721892939259 1loss/SRqqt2/assignment_loss‚0.585023533119615 2loss/FBt1/assignment_loss‚0.016961456727769015 /loss/FRt2/detection_loss‚1.6840445051648627 1loss/SRqqt2/detection_loss‚0.6988944248141835 1loss/SRqqt1/detection_loss‚0.7040855490104981 /loss/FBt1/detection_loss‚0.3924420018740695ÛR.ë ÷† 1loss/SRqqt1/detection_loss‚0.7040855490104981 /loss/FBt1/detection_loss‚0.3924420018740695 trainer/global_step‚3049 $ _timestamp‚1.784335190378266e+09 _runtime‚5350.963497324 /loss/FRt1/detection_loss‚0.5329966346416594 epoch‚7 _step‚250 /loss/FRt1/assignment_loss‚1.815671792414733 3loss/FBt2/assignment_loss‚0.0006135418826327717 /loss/FBt2/detection_loss‚0.9102748233292784 %loss/total_loss‚8.926061448261404 0loss/FRt2/assignment_loss‚1.0359809959881774 2loss/SRqqt1/assignment_loss‚0.5490721892939259 1loss/SRqqt2/assignment_loss‚0.585023533119615 2loss/FBt1/assignment_loss‚0.016961456727769015 /loss/FRt2/detection_loss‚1.6840445051648627 1loss/SRqqt2/detection_loss‚0.6988944248141835ú…¢ùDŸø‚(:” Ú–ëÒàÓäÈ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚187.686' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.537200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0býÛù‚(:Ð Ú–ëÒ¨àÄÉ disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda3.in‚ 0.47265625 disk.rbd0.out‚0 disk.dm-1.out‚ 194.04296875 disk.sda.in‚ 0.47265625 network.sent‚ 5.760016e+06 disk.dm-1.in‚ 0.47265625 disk.sda.out‚ 194.04296875 disk.drbd2165.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚34653.0576171875" disk.dm-0.out‚34653.0576171875 proc.memory.rssMB‚2899.25) proc.memory.percent‚17.69561767578125$ disk./.usageGB‚585.0968437194824 disk.sda3.out‚ 194.04296875 cpu‚25.531380723900714 disk.nvme0n1.in‚ 6576.5625% disk.nvme0n1.out‚34653.0576171875 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 6576.5625 proc.cpu.threads‚17 network.recv‚ 1.351338e+06* disk./.usagePercent‚39.266340110965544 disk.dm-0.in‚ 6576.5625$ memory_percent‚88.13462257385254 disk.rbd1.in‚0 disk.rbd1.out‚ 14.3671875( proc.memory.availableMB‚ 1944.0234375ëK@#Ÿú‚(:” é–ëÒ±ÌÇ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚217.657' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚43.531400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ£(àû‚(:Õ é–ëÒñ”ØÉ) proc.memory.availableMB‚ 1943.83984375 cpu‚25.530921195736255 disk.rbd1.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 6577.52734375% disk.nvme0c0n1.out‚34660.64453125# disk.nvme0n1.out‚34660.64453125$ disk./.usageGB‚585.0962562561035 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 194.26171875 disk.sda3.in‚ 0.47265625 proc.memory.rssMB‚2899.25) proc.memory.percent‚17.69561767578125 disk.dm-1.out‚ 194.26171875 disk.sda1.in‚0 disk.dm-0.out‚34660.64453125 disk.rbd1.out‚ 14.4296875 disk.sda.in‚ 0.47265625 disk.dm-0.in‚ 6577.52734375 disk.rbd0.in‚0$ memory_percent‚88.13574314117432 proc.cpu.threads‚17 network.sent‚ 5.823481e+06) disk./.usagePercent‚39.26630068580527 disk.dm-1.in‚ 0.47265625 disk.drbd2165.in‚0 disk.sda3.out‚ 194.26171875 network.recv‚ 1.35671e+06! disk.nvme0n1.in‚ 6577.52734375 disk.sda1.out‚0 disk.rbd0.out‚0䃞ü‚(:“ ø–ëÒ¶öæÇ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚325.553' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.11059999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…Ü@yñý‚(:æ ø–ëÒ•ëÉ) disk./.usagePercent‚39.26629223755664 disk.sda1.in‚0 disk.rbd1.in‚0 network.sent‚ 5.828339e+06 disk.dm-0.in‚ 6656.31640625* proc.memory.percent‚16.872811317443848 disk.sda1.out‚0 disk.sda.in‚ 0.47265625# proc.memory.rssMB‚ 2764.44140625 cpu‚25.530510525619775 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚34670.67724609375& disk.nvme0n1.out‚34670.67724609375 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 194.44140625 disk.rbd1.out‚ 14.4296875 network.recv‚ 1.360311e+06# disk.nvme0c0n1.in‚ 6656.31640625 disk.sda2.out‚0 disk.sda.out‚ 194.44140625 disk.drbd2165.out‚0$ disk./.usageGB‚585.0961303710938# disk.dm-0.out‚34670.67724609375! disk.nvme0n1.in‚ 6656.31640625 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 194.44140625$ memory_percent‚87.40875720977783 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2062.94921875"-Çíþ‚(:â ‡—ëÒ¬óÜÈ disk.sda1.in‚0 network.sent‚ 5.833419e+06" disk.dm-0.out‚34674.5830078125 disk.rbd1.out‚ 14.5078125 disk.sda.in‚ 0.47265625 disk.sda.out‚ 194.87109375 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 6656.31640625$ disk./.usageGB‚585.0960960388184 disk.sda2.in‚0# memory_percent‚87.4086856842041( proc.memory.availableMB‚ 2062.9609375! disk.nvme0n1.in‚ 6656.31640625 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 disk.dm-1.in‚ 0.47265625 proc.cpu.threads‚17* disk./.usagePercent‚39.266289933488835 disk.dm-0.in‚ 6656.31640625 disk.sda1.out‚0 disk.sda3.out‚ 194.87109375 disk.rbd1.in‚0# proc.memory.rssMB‚ 2764.44921875' disk.nvme0c0n1.out‚34674.5830078125% disk.nvme0n1.out‚34674.5830078125 disk.rbd0.out‚0 disk.dm-1.out‚ 194.87109375* proc.memory.percent‚16.872859001159668 cpu‚25.530068570495878 network.recv‚ 1.363568e+06 disk.drbd2165.out‚0 disk.sda2.out‚0ÛYดÿ‚(:‰ ‡—ëÒæÌäÈ gpu.0.gpu‚76.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚321.162' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.2324 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³ØÞ”€‚(:‰ –—ëÒРÑÇ gpu.0.gpu‚42.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚326.764' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.3528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ]sÕã‚(:Ø –—ëÒ—©ÔÉ network.sent‚ 5.838342e+06 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 6657.28125 proc.cpu.threads‚17* disk./.usagePercent‚39.266295309647056 disk.rbd1.out‚ 14.5078125 cpu‚25.529608751792107$ disk./.usageGB‚585.0961761474609% disk.nvme0n1.out‚34685.5693359375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 195.0859375 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 6657.28125) proc.memory.availableMB‚ 2062.96484375 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.47265625* proc.memory.percent‚16.872859001159668 network.recv‚ 1.367206e+06 disk.sda3.out‚ 195.0859375 disk.dm-1.in‚ 0.47265625' disk.nvme0c0n1.out‚34685.5693359375" disk.dm-0.out‚34685.5693359375$ memory_percent‚87.40866184234619 disk.nvme0n1.in‚ 6657.28125 disk.dm-1.out‚ 195.0859375# proc.memory.rssMB‚ 2764.44921875–\î‚‚(:‚ ¥—ëÒÜ“äÇ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚330.0' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.0 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06ˆÓ1郂(:Þ ¥—ëÒø¸¹É$ disk./.usageGB‚585.0962181091309* disk./.usagePercent‚39.266298125729925# disk.dm-0.out‚34693.81884765625 disk.rbd0.out‚0 disk.sda.in‚ 0.47265625# memory_percent‚87.4086856842041( disk.nvme0c0n1.out‚34693.81884765625! disk.nvme0n1.in‚ 6658.11328125 disk.rbd1.in‚0 disk.sda.out‚ 195.421875 network.recv‚ 1.370668e+06# disk.nvme0c0n1.in‚ 6658.11328125& disk.nvme0n1.out‚34693.81884765625 disk.rbd0.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 5.843314e+06 disk.sda1.out‚0 disk.rbd1.out‚ 14.5703125 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 195.421875 disk.dm-1.out‚ 195.421875 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2764.44921875* proc.memory.percent‚16.872859001159668 disk.dm-0.in‚ 6658.11328125 disk.sda1.in‚0 disk.dm-1.in‚ 0.47265625 disk.sda2.out‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 2062.9609375 cpu‚25.52911694320591 Y$î2„- _wandbruntime‚5433  _runtime‚5433…L5uÉ…Ã trainer/global_step‚3099 % _timestamp‚1.7843352724967086e+09 _runtime‚5433.082847562 _step‚251 ' lr-AdamW/pg1‚0.00046024936101290434 ' lr-AdamW/pg2‚0.00046024936101290434ƒ>×Î†È % _timestamp‚1.7843352724967086e+09 _runtime‚5433.082847562 _step‚251 ' lr-AdamW/pg1‚0.00046024936101290434 ' lr-AdamW/pg2‚0.00046024936101290434 trainer/global_step‚3099ûÔ.½¤2‡- _wandbruntime‚5433  _runtime‚5433- gï'ˆéO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚891597390573056 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚888685609459712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26805803 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚240736400896000 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16744646 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2913126502912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚891598762136064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26367013 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚891209533018386 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26367007 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚240736768987136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚888301363277824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26805803 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26815776 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _step‚252 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26367007 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚240736768987136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚888301363277824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26805803 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26815776 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _step‚252 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚891597390573056 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚888685609459712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26805803 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚240736400896000 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16744646 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2913126502912üféÑ¥2Š- _wandbruntime‚5434  _runtime‚5434rF€×ö&‹ðM M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53189494 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244572007776256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8797226 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244572408299520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚240813887303680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16749694 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16749930 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚888963636363264 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26824268 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚891877687908864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚891876316338688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚891486968234518 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 _step‚253 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53199487 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26375219 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26375213 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2914051545600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3758152904704 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53199487 _runtime‚5434.535824135 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26375213 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8797457 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚240814255394816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2912706148864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26824268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7952473 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚891488338262920 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2907723178518 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚888963636363264 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26814281 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7952468 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2909067778952 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434›9§*û&ŒõM PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26814281 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7952468 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2909067778952 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53189494 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244572007776256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8797226 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244572408299520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚240813887303680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16749694 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16749930 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚888963636363264 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26824268 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚891877687908864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚891876316338688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚891486968234518 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 _step‚253 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53199487 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26375219 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26375213 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2914051545600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3758152904704 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53199487 _runtime‚5434.535824135 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26375213 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8797457 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚240814255394816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2912706148864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26824268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7952473 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚891488338262920 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2907723178518 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚888963636363264 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚905543743009280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27231555 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚248427807391744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚905149859040294 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2954381425702 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54018841 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 % _timestamp‚1.7843353559274306e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2958097043968 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26777313 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚905543743009280 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17008776 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3816718020608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26777307 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚902585645965312 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8077075 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚905545114572288 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2959442433536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚902585672138752 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2958097043968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26777307 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚244611089371136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244610721280000 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2958097043968 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26777313 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚905543743009280 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17008776 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3816718020608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26777307 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚902585645965312 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8077075 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚905545114572288 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2959442433536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚902585672138752 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2958097043968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26777307 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚244611089371136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244610721280000 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚905543743009280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27231555 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚248427807391744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚905149859040294 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2954381425702 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54018841 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 % _timestamp‚1.7843353559274306e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27250020 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3817879706624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚905822668774912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚905428664284828 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2953960291626 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26785513 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27240033 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚902863672868864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27250020 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54035539 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 trainer/global_step‚3149 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚902863646695424 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚902863672868864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27240033 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26785519 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17013816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚905427294256426 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2959022079488è3ùe¢÷M F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚902863672868864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27240033 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26785519 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17013816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚905427294256426 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2959022079488 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚248506487917568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248506087394304 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚905824040345088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚902863646695424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26785519 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8079623 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚8 ÇM ˜20 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27250020 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3817879706624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚905822668774912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚905428664284828 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2953960291626 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26785513 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27240033 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚902863672868864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27250020 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54035539 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 trainer/global_step‚3149 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚902863646695424 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736¨dØ—2£- _wandbruntime‚5517  _runtime‚5517‹º¤€ /loss/FRt2/assignment_loss‚0.932703242778778 /loss/FRt2/detection_loss‚1.7873669998844464 1loss/SRqqt1/detection_loss‚0.7471071675154758 _step‚258 1loss/FBt1/assignment_loss‚0.05603120953919093 /loss/FRt1/detection_loss‚0.5179308483019249 .loss/FBt1/detection_loss‚0.452785898250897 /loss/FBt2/detection_loss‚0.7280215879935505 %loss/total_loss‚8.779481297654328 epoch‚7 0loss/FRt1/assignment_loss‚1.7268600217508574 3loss/SRqqt2/assignment_loss‚0.49505563118062546 0loss/FBt2/assignment_loss‚0.0724285139767904 1loss/SRqqt2/detection_loss‚0.7858018701441222 trainer/global_step‚3149 3loss/SRqqt1/assignment_loss‚0.47738830633766954 % _timestamp‚1.7843353574121344e+09 _runtime‚5517.997429237„Cbb ¥… 1loss/FBt1/assignment_loss‚0.05603120953919093 /loss/FRt1/detection_loss‚0.5179308483019249 .loss/FBt1/detection_loss‚0.452785898250897 /loss/FBt2/detection_loss‚0.7280215879935505 %loss/total_loss‚8.779481297654328 epoch‚7 0loss/FRt1/assignment_loss‚1.7268600217508574 3loss/SRqqt2/assignment_loss‚0.49505563118062546 0loss/FBt2/assignment_loss‚0.0724285139767904 1loss/SRqqt2/detection_loss‚0.7858018701441222 trainer/global_step‚3149 3loss/SRqqt1/assignment_loss‚0.47738830633766954 % _timestamp‚1.7843353574121344e+09 _runtime‚5517.997429237 /loss/FRt2/assignment_loss‚0.932703242778778 /loss/FRt2/detection_loss‚1.7873669998844464 1loss/SRqqt1/detection_loss‚0.7471071675154758 _step‚258‚NèÒ蟦‚(:” ÿ—ëÒáð¡È gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚136.974' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.394799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-¢´é§‚(:Þ ÿ—ëÒ˜˜àÉ network.recv‚ 1.392752e+06$ disk./.usageGB‚585.0967483520508 disk.sda.in‚ 0.47265625' disk.nvme0c0n1.out‚34732.3701171875 disk.drbd2165.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 2062.8046875 disk.sda.out‚ 197.05078125 disk.dm-0.in‚ 6844.71484375 disk.sda3.out‚ 197.05078125 disk.rbd0.out‚0) disk./.usagePercent‚39.26633371077719 disk.dm-1.out‚ 197.05078125" disk.dm-0.out‚34732.3701171875 disk.sda3.in‚ 0.47265625$ memory_percent‚87.40963935852051 disk.dm-1.in‚ 0.47265625 disk.drbd2165.out‚0 cpu‚25.52652253165803 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 6844.71484375% disk.nvme0n1.out‚34732.3701171875" proc.memory.rssMB‚ 2764.4609375 proc.cpu.threads‚17 network.sent‚ 5.931358e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 14.7890625! disk.nvme0n1.in‚ 6844.71484375( proc.memory.percent‚16.8729305267334È-s”¨‚(:‰ Ž˜ëÒ€ƒµÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚122.684' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 XY=ש‚(:Ì Ž˜ëÒ‡øÒÉ$ disk./.usageGB‚585.0964546203613# disk.nvme0n1.out‚34741.62890625 disk.rbd1.out‚ 14.7890625 cpu‚25.52600233257332 disk.dm-0.in‚ 6845.546875 disk.dm-0.out‚34741.62890625 disk.sda3.in‚ 0.47265625" proc.memory.rssMB‚ 2764.4609375) proc.memory.availableMB‚ 2062.76171875 network.recv‚ 1.397323e+06) disk./.usagePercent‚39.26631399819705 disk.rbd1.in‚0 disk.sda.out‚ 197.265625 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 6845.546875( proc.memory.percent‚16.8729305267334 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 197.265625$ memory_percent‚87.40990161895752 proc.cpu.threads‚17 disk.nvme0n1.in‚ 6845.546875 disk.rbd0.in‚0 disk.dm-1.in‚ 0.47265625 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0% disk.nvme0c0n1.out‚34741.62890625 network.sent‚ 5.99429e+06 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 197.265625øÐÁžª‚(:“ ˜ëÒâܧÇ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚297.28' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.455999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à 竂(:Ü ˜ëÒ¿ï÷È" proc.memory.rssMB‚ 2764.4609375 disk.sda2.in‚0 disk.sda.in‚ 0.47265625 disk.sda.out‚ 197.4765625 disk.dm-0.in‚ 6846.51171875* disk./.usagePercent‚39.266298125729925 disk.sda2.out‚0 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 6846.51171875 network.sent‚ 5.999314e+06 disk.rbd0.in‚0 disk.dm-1.out‚ 197.4765625' disk.nvme0c0n1.out‚34749.3193359375' proc.memory.availableMB‚ 2062.765625( proc.memory.percent‚16.8729305267334 cpu‚25.525535622911196$ disk./.usageGB‚585.0962181091309 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 197.4765625 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625! disk.nvme0n1.in‚ 6846.51171875 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚34749.3193359375 disk.rbd1.out‚ 14.9140625" disk.dm-0.out‚34749.3193359375 network.recv‚ 1.400923e+06$ memory_percent‚87.40987777709961¨¥¼¼”¬‚(:‰ ¬˜ëÒÒÊÞÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚114.437' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8874 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MLìã­‚(:Ø ¬˜ëÒáú¯É network.recv‚ 1.40444e+06 disk.rbd0.in‚0 disk.sda.in‚ 0.47265625" disk.dm-0.out‚34754.4736328125 disk.sda1.out‚0 disk.dm-1.in‚ 0.47265625' disk.nvme0c0n1.out‚34754.4736328125% disk.nvme0n1.out‚34754.4736328125 disk.sda3.in‚ 0.47265625 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda.out‚ 197.671875 disk.drbd2165.in‚0 disk.drbd2165.out‚0* disk./.usagePercent‚39.266293261586775 disk.sda2.in‚0 disk.dm-1.out‚ 197.671875 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 6927.11328125 disk.dm-0.in‚ 6927.11328125( proc.memory.availableMB‚ 2062.5546875 disk.sda3.out‚ 197.671875 cpu‚25.525105135368868 network.sent‚ 6.004367e+06$ disk./.usageGB‚585.0961456298828 disk.sda1.in‚0$ memory_percent‚87.41116523742676 proc.memory.rssMB‚ 2764.46875 disk.rbd1.out‚ 14.9140625! disk.nvme0n1.in‚ 6927.11328125) proc.memory.percent‚16.87297821044922 proc.cpu.threads‚17@Dðí”®‚(:‰ »˜ëÒñš¤Ç gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05V¸Šç¯‚(:Ü »˜ëÒ™¬ÇÉ disk.sda1.in‚0 disk.dm-1.out‚ 197.8671875 disk.drbd2165.out‚0$ memory_percent‚87.40994930267334) proc.memory.availableMB‚ 2062.75390625 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 6927.9453125 disk.rbd0.in‚0 disk.drbd2165.in‚0& disk.nvme0n1.out‚34761.81005859375 network.sent‚ 6.009345e+06 network.recv‚ 1.407995e+06$ disk./.usageGB‚585.0961647033691 disk.sda.in‚ 0.47265625 disk.sda.out‚ 197.8671875 proc.cpu.threads‚17 disk.dm-1.in‚ 0.47265625 disk.sda3.out‚ 197.8671875( disk.nvme0c0n1.out‚34761.81005859375# disk.dm-0.out‚34761.81005859375 disk.nvme0n1.in‚ 6927.9453125 disk.sda3.in‚ 0.47265625 disk.rbd1.out‚ 14.9921875 proc.memory.rssMB‚ 2764.46875 cpu‚25.524679471780914 disk.sda2.out‚0) disk./.usagePercent‚39.26629454162445 disk.rbd0.out‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 6927.9453125) proc.memory.percent‚16.872978210449228¥ò”°‚(:‰ ʘëÒ€§öÇ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚255.046' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.0092 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èÛ*뱂(:à ʘëÒÔÅÉÉ) proc.memory.percent‚16.87297821044922$ disk./.usageGB‚585.0962181091309& disk.nvme0n1.out‚34772.10498046875 cpu‚25.524257504426064 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 6928.91015625 proc.memory.rssMB‚ 2764.46875 disk.sda3.in‚ 0.47265625 disk.rbd1.out‚ 14.9921875( disk.nvme0c0n1.out‚34772.10498046875# memory_percent‚87.4100923538208 network.recv‚ 1.4116e+06 disk.rbd0.out‚0 disk.sda.out‚ 198.23046875# disk.nvme0c0n1.in‚ 6928.91015625# disk.dm-0.out‚34772.10498046875) proc.memory.availableMB‚ 2062.73046875 network.sent‚ 6.014352e+06 disk.sda3.out‚ 198.23046875 disk.sda.in‚ 0.47265625 disk.dm-1.out‚ 198.23046875 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚39.266298125729925 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 0.47265625! disk.nvme0n1.in‚ 6928.910156250àB2²- _wandbruntime‚5600  _runtime‚5600n‡™édzÁ _runtime‚5600.562175126 _step‚259 & lr-AdamW/pg1‚0.0004586002269985516 & lr-AdamW/pg2‚0.0004586002269985516 trainer/global_step‚3199 % _timestamp‚1.7843354399771037e+09æ¿iÌ´Æ trainer/global_step‚3199 % _timestamp‚1.7843354399771037e+09 _runtime‚5600.562175126 _step‚259 & lr-AdamW/pg1‚0.0004586002269985516 & lr-AdamW/pg2‚0.0004586002269985516ƒËÄŠX2µ- _wandbruntime‚5600  _runtime‚5600<6Søï'¶éO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54844914 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17272870 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3005758364160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3876741137408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚916089566523392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8204230 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2999273938488 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27187607 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _runtime‚5600.563774347 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54854893 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚919491467008512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚252362150892544 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9068884 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3004412974592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27187613 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27657307 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248485409755136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27667280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚252361750362112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚919088815033912 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚919491467008512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27657307 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8204225 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚919090185062202 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚916485682470912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3876708698112 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚919490095445504 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚916485708644352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27187613 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚916485708644352 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3005758364160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3004412974592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚916485682470912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27667280 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚248485041664000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7843354399783242e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚260 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640º‰]ª¼·îO O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _runtime‚5600.563774347 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54854893 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚919491467008512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚252362150892544 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9068884 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3004412974592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27187613 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27657307 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248485409755136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27667280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚252361750362112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚919088815033912 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚919491467008512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27657307 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8204225 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚919090185062202 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚916485682470912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3876708698112 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚919490095445504 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚916485708644352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27187613 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚916485708644352 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3005758364160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3004412974592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚916485682470912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27667280 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚248485041664000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7843354399783242e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚260 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54844914 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17272870 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3005758364160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3876741137408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚916089566523392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8204230 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2999273938488 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27187607 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820„™;¦42¸- _wandbruntime‚5601  _runtime‚5601eNVù&¹óM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27195813 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27675772 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚252440435277824 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27195813 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17278404 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8206768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3006683406848 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27195819 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚916763683201024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚919367620278334 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27665785 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252440835801088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚916367422873600 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54861598 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚261 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚919769021211136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚916763709374464 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54871591 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3005338010112 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚248562528071680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3001542005168 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27675772 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9071631 _runtime‚5601.972330595 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3006683406848 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27665785 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 trainer/global_step‚3199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17278164 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚916763683201024 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54871591 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3877939638272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54861598 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3000197404734 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚916763709374464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚916367448301568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27195819 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3006683406848 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27665785 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 trainer/global_step‚3199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17278164 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚916763683201024 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54871591 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3877939638272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54861598 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3000197404734 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚916763709374464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚916367448301568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27195819 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27195813 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17278404 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8206768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3006683406848 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27195819 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚916763683201024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚919367620278334 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27665785 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252440835801088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚916367422873600 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54861598 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚261 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚919769021211136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚916763709374464 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54871591 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3005338010112 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚248562528071680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3001542005168 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627…PŸH2»- _wandbruntime‚5602  _runtime‚5602" K¼€ 2loss/SRqqt1/assignment_loss‚0.6064728711526727 1loss/SRqqt1/detection_loss‚0.7277892398382044 /loss/FBt1/detection_loss‚0.7366771362103464 /loss/FBt2/detection_loss‚0.9139896833309622 $loss/total_loss‚9.44870299181099 _step‚262 0loss/FRt1/assignment_loss‚1.9073862941353232 /loss/FRt2/assignment_loss‚0.994171302228391 /loss/FRt2/detection_loss‚1.6503246455821021 1loss/SRqqt2/detection_loss‚0.7421270494374855 $ _timestamp‚1.784335441466638e+09 _runtime‚5602.051742876 2loss/SRqqt2/assignment_loss‚0.5554530857992914 2loss/FBt2/assignment_loss‚0.004611053846232087 /loss/FRt1/detection_loss‚0.5884144737332269 epoch‚7 trainer/global_step‚3199 2loss/FBt1/assignment_loss‚0.021286156516752308yëî ½… 2loss/SRqqt2/assignment_loss‚0.5554530857992914 2loss/FBt2/assignment_loss‚0.004611053846232087 /loss/FRt1/detection_loss‚0.5884144737332269 epoch‚7 trainer/global_step‚3199 2loss/FBt1/assignment_loss‚0.021286156516752308 2loss/SRqqt1/assignment_loss‚0.6064728711526727 1loss/SRqqt1/detection_loss‚0.7277892398382044 /loss/FBt1/detection_loss‚0.7366771362103464 /loss/FBt2/detection_loss‚0.9139896833309622 $loss/total_loss‚9.44870299181099 _step‚262 0loss/FRt1/assignment_loss‚1.9073862941353232 /loss/FRt2/assignment_loss‚0.994171302228391 /loss/FRt2/detection_loss‚1.6503246455821021 1loss/SRqqt2/detection_loss‚0.7421270494374855 $ _timestamp‚1.784335441466638e+09 _runtime‚5602.051742876†)õ˜kž¾‚(:“ Ù˜ëÒåœâÇ gpu.0.gpu‚97.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚329.465' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.89299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙÂ:è¿‚(:Ý Ù˜ëÒÑÀüÈ" disk.dm-0.out‚34776.5166015625! disk.nvme0n1.in‚ 6928.91015625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 15.0546875 disk.sda.out‚ 198.4140625 disk.dm-0.in‚ 6928.91015625* proc.memory.percent‚17.391228675842285 disk.sda3.out‚ 198.4140625 disk.dm-1.out‚ 198.4140625 disk.drbd2165.in‚0% disk.nvme0n1.out‚34776.5166015625' proc.memory.availableMB‚ 1976.734375 disk.rbd0.out‚0 network.recv‚ 1.41507e+06 disk.sda3.in‚ 0.47265625 disk.drbd2165.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.26630145382787# proc.memory.rssMB‚ 2849.37890625 disk.sda2.in‚0 network.sent‚ 6.019321e+06$ disk./.usageGB‚585.0962677001953# disk.nvme0c0n1.in‚ 6928.91015625 cpu‚25.526904910600628 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625$ memory_percent‚87.93497085571289' disk.nvme0c0n1.out‚34776.5166015625 0åÀ‚(:… è˜ëÒѤÈÈ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß­²0èÁ‚(:Ý è˜ëÒöÍÂÉ disk.sda3.in‚ 0.47265625* disk./.usagePercent‚39.266299405767604 disk.rbd1.in‚0 disk.sda.out‚ 198.6484375" disk.nvme0c0n1.in‚ 7029.8359375 disk.nvme0n1.in‚ 7029.8359375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 disk.rbd1.out‚ 15.0546875) proc.memory.availableMB‚ 1959.37890625 proc.cpu.threads‚17 network.sent‚ 6.082249e+06" disk.dm-0.out‚34787.9189453125 disk.sda3.out‚ 198.6484375 disk.rbd0.in‚0 cpu‚25.533686401991222 network.recv‚ 1.419545e+06% disk.nvme0n1.out‚34787.9189453125 disk.sda2.out‚0 disk.sda.in‚ 0.47265625 disk.sda1.in‚0 disk.dm-1.out‚ 198.6484375$ memory_percent‚88.04090023040771) proc.memory.percent‚17.49589443206787 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚34787.9189453125 disk.dm-0.in‚ 7029.8359375 disk.sda1.out‚0# proc.memory.rssMB‚ 2866.52734375$ disk./.usageGB‚585.0962371826172 disk.drbd2165.in‚0n*¯h2Â- _wandbruntime‚5629  _runtime‚5629¨ÓœÑî{ ("Purity/1FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9455226063728333 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9897704124450684 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666865348816 jet/accuracy_1_of_5‚1 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09625786542892456 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2946200668811798 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5682449340820312 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974368810653687 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8949843645095825 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.406604528427124 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.32940682768821716 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44212961196899414 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9869047999382019 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0637969970703125 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8112325072288513 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 particle/accuracy_4_of_5‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9928945899009705 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6476653218269348 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 +jet/accuracy_2_of_2‚0.15244144201278687 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_2_of_3‚0.5646593570709229 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955652952194214 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34630483388900757 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_2_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974368810653687 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0694323256611824 *jet/accuracy_1_of_1‚0.3448503911495209 particle/accuracy_2_of_3‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6078790426254272 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5834982991218567 +jet/accuracy_3_of_4‚0.09278637915849686 particle/accuracy_2_of_2‚1 'particle/f_score‚0.6419568061828613 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.32692039012908936 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6405758857727051 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6192274689674377 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4639910161495209 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6423501968383789 ,jet/accuracy_4_of_4‚0.011098379269242287 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9977869391441345 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.30640584230422974 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7274744510650635 _step‚263 :"Purity/*FRt*SRqqt2FBt/SRqqt_SϦßã-purity‚0.15740740299224854 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6955513954162598 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6582100987434387 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10147324949502945 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.32890090346336365 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.647903561592102 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +particle/sensitivity‚0.6231657862663269 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9941471815109253 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6137019991874695 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6274107694625854 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6608045101165771 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8112325072288513 particle/accuracy_3_of_3‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48577970266342163 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6899447441101074 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9813995361328125 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.32890090346336365 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 particle/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965608716011047 trainer/global_step‚3199 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5742292404174805 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6329396963119507 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9872037172317505 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.606240451335907 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7453703284263611 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.78125 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.304496169090271 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9897873401641846 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35443753004074097 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.33699268102645874 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7661784291267395 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_4‚1 6validation_average_jet_accuracy‚0.3842819631099701 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6390394568443298 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 particle/accuracy_1_of_3‚1 +particle/specificity‚0.9129639863967896 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6742918491363525 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3384156823158264 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6755809187889099 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.053131796419620514 *jet/accuracy_2_of_4‚0.8108271360397339 particle/accuracy_1_of_1‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5711463689804077 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2968187630176544 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5784756541252136 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10112551599740982 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9827294945716858 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3369148075580597 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_5_of_5‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8214285969734192 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6282329559326172 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961844086647034 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8244143128395081 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9755765199661255 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5788981914520264 *jet/accuracy_1_of_3‚0.9308825135231018 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.33744627237319946 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.23050381243228912 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48473915457725525 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 /particle/accuracy_3_of_4‚0.9979296326637268 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7"Purity/0FRt1SRqqt*FBt/event_purity‚0.57023686170578 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 % _timestamp‚1.7843354693749502e+09 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.25628507137298584 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3038978576660156 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6380208134651184 jet/accuracy_3_of_5‚1 epoch‚7 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8398337364196777 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.15740740299224854 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18209931254386902 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9877882599830627 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9933311939239502 *jet/accuracy_1_of_2‚0.7974233031272888 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09612046927213669 (particle/accuracy‚0.8507254719734192 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8678421974182129 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7540627717971802 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6894537210464478 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5711463689804077 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7479897141456604 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.25571274757385254 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6616806983947754 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9074073433876038 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.24740223586559296 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7374920845031738 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08250126987695694 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3790043294429779 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4185701012611389 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.27702969312667847 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5358332991600037 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7453703284263611 *jet/accuracy_1_of_4‚0.9665131568908691 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3408425450325012 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.555081844329834 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9286912083625793 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7191647291183472 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 )loss/val_total_loss‚8.739524682824172 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8395314812660217 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.611142098903656 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.32002589106559753 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.24460159242153168 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4851635992527008 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3193284869194031 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9181712865829468 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9945071339607239 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2424510270357132 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7175543308258057 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 _runtime‚5629.960708253 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.632454514503479 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.836319088935852 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14416667819023132 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.13409548997879028 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9971776008605957 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.12307843565940857 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.28924044966697693 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07543574273586273 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32043886184692383 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980305433273315 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8382842540740967 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5844887495040894 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.32664045691490173 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08250126987695694 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18551619350910187 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04123660549521446 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7228447198867798 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8681628108024597 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272ßQZ5¹=ij{ 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3408425450325012 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.555081844329834 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9286912083625793 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7191647291183472 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 )loss/val_total_loss‚8.739524682824172 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8395314812660217 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.611142098903656 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.32002589106559753 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.24460159242153168 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4851635992527008 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3193284869194031 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9181712865829468 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9945071339607239 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2424510270357132 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7175543308258057 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 _runtime‚5629.960708253 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.632454514503479 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.836319088935852 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14416667819023132 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.13409548997879028 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9971776008605957 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.12307843565940857 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.28924044966697693 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07543574273586273 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32043886184692383 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980305433273315 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8382842540740967 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5844887495040894 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.32664045691490173 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08250126987695694 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18551619350910187 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04123660549521446 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7228447198867798 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8681628108024597 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9455226063728333 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9897704124450684 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666865348816 jet/accuracy_1_of_5‚1 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09625786542892456 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2946200668811798 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5682449340820312 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974368810653687 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8949843645095825 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.406604528427124 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.32940682768821716 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44212961196899414 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9869047999382019 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0637969970703125 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8112325072288513 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 particle/accuracy_4_of_5‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9928945899009705 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6476653218269348 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 +jet/accuracy_2_of_2‚0.15244144201278687 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_2_of_3‚0.5646593570709229 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955652952194214 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34630483388900757 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_2_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974368810653687 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0694323256611824 *jet/accuracy_1_of_1‚0.3448503911495209 particle/accuracy_2_of_3‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6078790426254272 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5834982991218567 +jet/accuracy_3_of_4‚0.09278637915849686 particle/accuracy_2_of_2‚1 'particle/f_score‚0.6419568061828613 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.32692039012908936 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6405758857727051 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6192274689674377 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4639910161495209 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6423501968383789 ,jet/accuracy_4_of_4‚0.011098379269242287 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9977869391441345 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.30640584230422974 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7274744510650635 _step‚263 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.15740740299224854 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6955513954162598 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6582100987434387 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10147324949502945 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.32890090346336365 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.647903561592102 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +particle/sensitivity‚0.6231657862663269 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9941471815109253 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6137019991874695 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6274107694625854 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6608045101165771 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8112325072288513 particle/accuracy_3_of_3‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48577970266342163 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6899447441101074 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9813995361328125 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.32890090346336365 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 particle/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965608716011047 trainer/global_step‚3199 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5742292404174805 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6329396963119507 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9872037172317505 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.606240451335907 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7453703284263611 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.78125 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.304496169090271 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9897873401641846 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35443753004074097 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.33699268102645874 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7661784291267395 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_4‚1 6validation_average_jet_accuracy‚0.3842819631099701 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6390394568443298 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 particle/accuracy_1_of_3‚1 +particle/specificity‚0.9129639863967896 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6742918491363525 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3384156823158264 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6755809187889099 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.053131796419620514 *jet/accuracy_2_of_4‚0.8108271360397339 particle/accuracy_1_of_1‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5711463689804077 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2968187630176544 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5784756541252136 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10112551599740982 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9827294945716858 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3369148075580597 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_5_of_5‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8214285969734192 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6282329559326172 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961844086647034 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8244143128395081 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9755765199661255 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5788981914520264 *jet/accuracy_1_of_3‚0.9308825135231018 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.33744627237319946 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.23050381243228912 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48473915457725525 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 /particle/accuracy_3_of_4‚0.9979296326637268 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7"Purity/0FRt1SRqqt*FBt/event_purity‚0.57023686170578 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 % _timestamp‚1.7843354693749502e+09 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.25628507137298584 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3038978576660156 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6380208134651184 jet/accuracy_3_of_5‚1 epoch‚7 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8398337364196777 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.15740740299224854 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18209931254386902 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9877882599830627 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9933311939239502 *jet/accuracy_1_of_2‚0.7974233031272888 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09612046927213669 (particle/accuracy‚0.8507254719734192 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8678421974182129 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7540627717971802 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6894537210464478 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5711463689804077 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7479897141456604 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.25571274757385254 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6616806983947754 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9074073433876038 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.24740223586559296 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7374920845031738 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08250126987695694 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3790043294429779 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4185701012611389 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.27702969312667847 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5358332991600037 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7453703284263611 *jet/accuracy_1_of_4‚0.9665131568908691‡ìXs”Å‚(:‰ ÷˜ëÒ®ŒÈ gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚286.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.2526 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`© —êÆ‚(:ß ÷˜ëÒÈé¦É disk.rbd1.in‚0 disk.rbd1.out‚ 15.1328125' disk.nvme0c0n1.out‚35765.7099609375" disk.dm-0.out‚35765.7099609375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 7030.80078125 disk.rbd0.out‚0" proc.memory.rssMB‚ 2870.0234375 disk.dm-1.out‚ 198.81640625# disk.nvme0c0n1.in‚ 7030.80078125 cpu‚25.532628465310783 disk.drbd2165.in‚0% disk.nvme0n1.out‚35765.7099609375 disk.sda3.in‚ 0.47265625 network.sent‚ 6.087271e+06 network.recv‚ 1.423157e+06) disk./.usagePercent‚39.26653390866897 disk.sda.out‚ 198.81640625 disk.sda2.out‚0' proc.memory.availableMB‚ 2408.265625 proc.cpu.threads‚17 disk.dm-0.in‚ 7030.80078125 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.51723289489746$ disk./.usageGB‚585.0997314453125 disk.dm-1.in‚ 0.47265625 disk.sda3.out‚ 198.81640625$ memory_percent‚85.30111312866211 disk.sda.in‚ 0.47265625N™fãÇ‚(:Ø †™ëÒé¦ØÉ disk.sda2.out‚0 network.sent‚ 6.145061e+06# disk./.usageGB‚585.100040435791 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625( disk.nvme0c0n1.out‚35770.15673828125 disk.rbd1.in‚0) proc.memory.availableMB‚ 2407.75390625* proc.memory.percent‚17.519831657409668! disk.nvme0n1.in‚ 7030.80078125 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.sda3.out‚199.3125# proc.memory.rssMB‚ 2870.44921875 proc.cpu.threads‚17& disk.nvme0n1.out‚35770.15673828125 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0 disk.dm-0.in‚ 7030.80078125 disk.dm-1.out‚199.3125 disk.sda.out‚199.3125 network.recv‚ 1.428047e+06) disk./.usagePercent‚39.26655464527924 disk.sda2.in‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 7030.80078125# disk.dm-0.out‚35770.15673828125 cpu‚25.53218652648038 disk.sda1.in‚0 disk.rbd1.out‚ 15.1328125 disk.rbd0.in‚0$ memory_percent‚85.30423641204834yd¿«žÈ‚(:“ †™ëÒ¶è”Ë gpu.0.gpu‚94.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚281.143' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.22859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éR@ÙŸÉ‚(:” •™ëÒÊÙâÇ gpu.0.gpu‚84.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚256.933' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.386599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VøpèÊ‚(:Ý •™ëÒÏÔëÉ disk.sda1.out‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 2407.0078125 network.sent‚ 6.149924e+06 disk.dm-1.in‚ 0.47265625 disk.rbd1.out‚ 15.1328125 disk.sda.out‚ 199.7265625 disk.drbd2165.in‚0# disk.dm-0.out‚36746.80322265625$ disk./.usageGB‚585.0962562561035 disk.sda2.in‚0 disk.sda.in‚ 0.47265625 network.recv‚ 1.431464e+06& disk.nvme0n1.out‚36746.80322265625 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 7031.6328125( disk.nvme0c0n1.out‚36746.80322265625 disk.rbd0.out‚0 disk.rbd1.in‚0) proc.memory.percent‚17.52331256866455 proc.cpu.threads‚17 disk.nvme0n1.in‚ 7031.6328125 disk.sda3.out‚ 199.7265625 disk.sda2.out‚0 cpu‚25.53267097360187) disk./.usagePercent‚39.26630068580527# proc.memory.rssMB‚ 2871.01953125 disk.sda1.in‚0 disk.dm-1.out‚ 199.7265625 disk.dm-0.in‚ 7031.6328125$ memory_percent‚85.30879020690918àH£'”Ë‚(:‰ ¤™ëÒ·ÇáÇ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚331.111' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.2222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ§õÌ‚(:Ù ¤™ëÒÇ×âÉ disk.dm-1.out‚ 200.03125 proc.cpu.threads‚17 disk.drbd2165.in‚0" disk.dm-0.out‚36756.1474609375 disk.sda2.out‚0 disk.sda.out‚ 200.03125* disk./.usagePercent‚39.266293773601845 disk.sda3.in‚ 0.47265625 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 network.sent‚ 6.154946e+06# disk.nvme0c0n1.in‚ 7117.74609375' disk.nvme0c0n1.out‚36756.1474609375% disk.nvme0n1.out‚36756.1474609375$ memory_percent‚85.31186580657959# proc.memory.rssMþçîàB‚ 2871.48046875) proc.memory.availableMB‚ 2406.50390625 network.recv‚ 1.435006e+06$ disk./.usageGB‚585.0961532592773 disk.dm-0.in‚ 7117.74609375 disk.sda3.out‚ 200.03125 disk.rbd0.in‚0 disk.rbd1.out‚ 15.2578125 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0) proc.memory.percent‚17.52612590789795 cpu‚25.53306745410133! disk.nvme0n1.in‚ 7117.74609375 disk.rbd0.out‚0Õ¿žÍ‚(:“ ³™ëÒÂÊÛÇ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚333.764' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.75280000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ“b£å΂(:Ú ³™ëÒèùØÉ proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.out‚ 200.21484375 disk.sda3.out‚ 200.21484375) disk./.usagePercent‚39.26629633367719 disk.dm-0.in‚ 7117.74609375 proc.memory.rssMB‚ 2872.21875 cpu‚25.53350548035226 disk.sda1.out‚0) proc.memory.availableMB‚ 2405.73046875 network.recv‚ 1.438355e+06 disk.dm-1.in‚ 0.47265625" disk.dm-0.out‚36760.2978515625 disk.sda3.in‚ 0.47265625) proc.memory.percent‚17.53063201904297 disk.rbd1.in‚0 disk.sda.out‚ 200.21484375' disk.nvme0c0n1.out‚36760.2978515625% disk.nvme0n1.out‚36760.2978515625 disk.sda1.in‚0 disk.rbd1.out‚ 15.2578125 disk.drbd2165.out‚0 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 7117.74609375 disk.rbd0.in‚0# memory_percent‚85.3165864944458 disk.rbd0.out‚0 network.sent‚ 6.16003e+06" disk./.usageGB‚585.09619140625 disk.sda2.in‚0! disk.nvme0n1.in‚ 7117.74609375ÌiB ”Ï‚(:‰ ™ëÒ°øòÈ gpu.0.gpu‚83.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚336.569' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.3138 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐóÞЂ(:Ó ™ëÒ±²°Ê$ disk./.usageGB‚585.0960922241211 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 7118.578125 proc.cpu.threads‚17) proc.memory.percent‚17.53082275390625 cpu‚25.53402129447252 network.recv‚ 1.441155e+06 disk.rbd1.out‚ 15.3984375 disk.rbd0.in‚0) proc.memory.availableMB‚ 2406.20703125* disk./.usagePercent‚39.266289677481296 disk.sda2.out‚0 disk.sda.in‚ 0.47265625 disk.sda.out‚ 200.515625 disk.drbd2165.in‚0 disk.dm-0.in‚ 7118.578125 disk.dm-1.in‚ 0.47265625$ memory_percent‚85.31367778778076 proc.memory.rssMB‚2872.25 disk.rbd1.in‚0 disk.sda3.out‚ 200.515625 disk.sda3.in‚ 0.47265625 network.sent‚ 6.164869e+06 disk.sda2.in‚0( disk.nvme0c0n1.out‚36771.37939453125# disk.dm-0.out‚36771.37939453125 disk.nvme0n1.in‚ 7118.578125& disk.nvme0n1.out‚36771.37939453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 200.515625`U0î2Ñ- _wandbruntime‚5718  _runtime‚5718Í·ñîÇÒÁ _step‚264 & lr-AdamW/pg1‚0.0004577546920062249 & lr-AdamW/pg2‚0.0004577546920062249 trainer/global_step‚3249 % _timestamp‚1.7843355575429914e+09 _runtime‚5718.12824503971šÌÓÆ _step‚264 & lr-AdamW/pg1‚0.0004577546920062249 & lr-AdamW/pg2‚0.0004577546920062249 trainer/global_step‚3249 % _timestamp‚1.7843355575429914e+09 _runtime‚5718.128245039ˆüÌ›12Ô- _wandbruntime‚5718  _runtime‚5718QÞz¦ï'ÕéO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚256351838465024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚937270271930880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚934213700411392 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚252407842385920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3943961542656 _runtime‚5718.129574531 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚934213726584832 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚256351435837440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27729919 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚937271643493888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚937271643493888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3943996079104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3056571519488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28152019 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚934213726584832 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28142046 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843355575443752e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚937270271930880 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3057916909056 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27729913 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3056571519488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9242408 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚936862249148160 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚933809564364800 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚66Á ô'ÖîO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚937270271930880 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3057916909056 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27729913 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3056571519488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9242408 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚936862249148160 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚933809564364800 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚256351838465024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚937270271930880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚934213700411392 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚252407842385920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3943961542656 _runtime‚5718.129574531 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚934213726584832 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚256351435837440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27729919 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚937271643493888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚937271643493888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3943996079104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3056571519488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28152019 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚934213726584832 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28142046 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843355575443752e+09‰X°bi2×- _wandbruntime‚5719  _runtime‚5719[ÁªÄ ØòM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3053608249718 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3052263649284 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3058841951744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27738119 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚934491701141504 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3057496555008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55898636 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚937549197696512 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28150524 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 $ _timestamp‚1.784335558954767e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚937550569266688 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚934491727314944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚256430123439104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5719.540343432 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚934491727314944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8342841 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3945197265920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚934491701141504 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17588029 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28160511 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55888643 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚934087420715008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17587783 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55888643 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3058841951744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚937139684364292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27738125 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252485328793600 C=DeviceStatsMonitor.on_train_batch_end/oversizö·çÐ4e_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚934087446142976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚937550569266688 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28160511 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9244947 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256430526059520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚252484960702464 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55898636 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627â­´ý&Ù÷M TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28160511 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55888643 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚934087420715008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17587783 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55888643 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3058841951744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚937139684364292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27738125 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252485328793600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚934087446142976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚937550569266688 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28160511 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9244947 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256430526059520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚252484960702464 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55898636 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3053608249718 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3052263649284 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3058841951744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27738119 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚934491701141504 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3057496555008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55898636 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚937549197696512 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28150524 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 $ _timestamp‚1.784335558954767e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚937550569266688 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚934491727314944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚256430123439104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5719.540343432 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚934491727314944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8342841 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3945197265920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚934491701141504 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17588029 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ŠÚD.2Ú- _wandbruntime‚5719  _runtime‚5719'õ¬Ûÿ _step‚267 1loss/SRqqt1/detection_loss‚0.7302831767592579 /loss/FBt2/detection_loss‚0.8726442653783842 trainer/global_step‚3249 0loss/FRt2/assignment_loss‚0.9005305060256081 1loss/FBt2/assignment_loss‚0.02773881436765853 /loss/FRt2/detection_loss‚1.6590016549077422 % _timestamp‚1.7843355590282893e+09 /loss/FRt1/assignment_loss‚1.801489852073688 3loss/SRqqt1/assignment_loss‚0.46061849150538653 2loss/SRqqt2/assignment_loss‚0.5781176047437245 #loss/total_loss‚8.8963515024614 _runtime‚5719.613524679 1loss/FBt1/assignment_loss‚0.00499779786605443 /loss/FRt1/detection_loss‚0.5561333274216265 1loss/SRqqt2/detection_loss‚0.7011162849773283 /loss/FBt1/detection_loss‚0.6036797264349405 epoch‚8,Lþ Ü„ 0loss/FRt2/assignment_loss‚0.9005305060256081 1loss/FBt2/assignment_loss‚0.02773881436765853 /loss/FRt2/detection_loss‚1.6590016549077422 % _timestamp‚1.7843355590282893e+09 /loss/FRt1/assignment_loss‚1.801489852073688 3loss/SRqqt1/assignment_loss‚0.46061849150538653 2loss/SRqqt2/assignment_loss‚0.5781176047437245 #loss/total_loss‚8.8963515024614 _runtime‚5719.613524679 1loss/FBt1/assignment_loss‚0.00499779786605443 /loss/FRt1/detection_loss‚0.5561333274216265 1loss/SRqqt2/detection_loss‚0.7011162849773283 /loss/FBt1/detection_loss‚0.6036797264349405 epoch‚8 _step‚267 1loss/SRqqt1/detection_loss‚0.7302831767592579 /loss/FBt2/detection_loss‚0.8726442653783842 trainer/global_step‚3249‹”àX žÝ‚(:“ Ñ™ëÒ÷ÖßÈ gpu.0.gpu‚62.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚225.45' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚45.089999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´‰©|ÐÞ‚(:Å Ñ™ëÒ›Î›Ê disk.nvme0n1.in‚ 7118.578125 disk.sda2.out‚0 proc.memory.rssMB‚2872.25 network.sent‚ 6.169101e+06 disk.sda.out‚ 200.78125# disk.nvme0n1.out‚36775.44140625 disk.rbd0.out‚0 disk.sda2.in‚0 cpu‚25.53454226338199 disk.sda1.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 2405.9765625 disk.sda3.in‚ 0.47265625 disk.dm-1.out‚ 200.78125 disk.rbd1.out‚ 15.3984375 disk.dm-0.in‚ 7118.578125) proc.memory.percent‚17.53082275390625 proc.cpu.threads‚17) disk./.usagePercent‚39.26627943717993 disk.dm-1.in‚ 0.47265625 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0% disk.nvme0c0n1.out‚36775.44140625 disk.dm-0.out‚36775.44140625$ memory_percent‚85.31508445739746 network.recv‚ 1.444491e+06 disk.sda3.out‚ 200.78125$ disk./.usageGB‚585.0959396362305 disk.rbd1.in‚0 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 7118.578125 disk.sda1.out‚0Ra±íß‚(:… à™ëÒ迵È gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚293.8' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.76 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!üã™äà‚(:Ù à™ëÒæîåÉ disk.rbd1.out‚ 15.4609375 disk.sda.in‚ 0.47265625 cpu‚25.534846301663833 network.recv‚ 1.449226e+06$ disk./.usageGB‚585.0958938598633 disk.drbd2165.out‚0 disk.dm-1.out‚ 200.9765625 disk.sda.out‚ 200.9765625 disk.drbd2165.in‚0) proc.memory.percent‚17.53082275390625( disk.nvme0c0n1.out‚36786.57470703125# disk.dm-0.out‚36786.57470703125 disk.sda2.in‚0& disk.nvme0n1.out‚36786.57470703125 disk.sda1.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 2405.96875 network.sent‚ 6.234293e+06) disk./.usagePercent‚39.26627636508952 disk.dm-1.in‚ 0.47265625! disk.nvme0n1.in‚ 7195.41015625 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 200.9765625 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 7195.41015625 disk.sda1.out‚0$ memory_percent‚85.31513214111328 proc.memory.rssMB‚2872.25# disk.nvme0c0n1.in‚ 7195.41015625 proc.cpu.threads‚17à¾Ùžá‚(:“ ï™ëÒ†Ê«Ç gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚272.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.565999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ªÞâ‚(:Ó ï™ëÒÄ¿±É' disk.nvme0c0n1.out‚36797.2509765625 disk.sda1.out‚0# memory_percent‚85.3151798248291 disk.dm-1.in‚ 0.47265625 network.sent‚ 6.237358e+06 disk.sda3.in‚ 0.47265625 disk.sda.in‚ 0.47265625 network.recv‚ 1.452952e+06) disk./.usagePercent‚39.26623693992924 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 203.0390625" disk.nvme0c0n1.in‚ 7196.2421875" disk.dm-0.out‚36797.2509765625% disk.nvme0n1.out‚36797.2509765625 cpu‚25.53527830746299 disk.drbd2165.out‚0 disk.dm-0.in‚ 7196.2421875 disk.dm-1.out‚ 203.0390625$ disk./.usageGB‚585.0953063964844 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 proc.memory.rssMB‚2872.25 disk.sda3.out‚ 203.0390625 disk.drbd2165.in‚0 disk.rbd1.out‚ 15.4609375 disk.nvme0n1.in‚ 7196.2421875( proc.memory.availableMB‚ 2405.9609375) proc.memory.percent‚17.53082275390625c÷³Vžã‚(:“ þ™ëÒ„‚µÇ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚260.765' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.15299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úÆ„Þä‚(:Ó þ™ëÒ’îÃÉ proc.cpu.threads‚17% disk.nvme0c0n1.out‚36804.25390625 disk.sda2.in‚0) proc.memory.percent‚17.53082275390625 cpu‚25.535746086024034 disk.nvme0n1.in‚ 7196.2421875 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625" disk.nvme0c0n1.in‚ 7196.2421875 network.recv‚ 1.456199e+06$ disk./.usageGB‚585.0958290100098 disk.drbd2165.in‚0 disk.rbd0.in‚0$ memory_percent‚85.31515598297119) disk./.usagePercent‚39.26627201296144 disk.dm-1.in‚ 0.47265625 disk.rbd1.out‚ 15.5390625 disk.drbd2165.out‚0 disk.dm-0.in‚ 7196.2421875 disk.sda1.in‚0) proc.memory.availableMB‚ 2405.96484375 disk.dm-1.out‚ 203.69140625 disk.rbd1.in‚0 disk.sda.out‚ 203.69140625 disk.dm-0.out‚36804.25390625# disk.nvme0n1.out‚36804.25390625 disk.sda3.out‚ 203.69140625 proc.memory.rssMB‚2872.25 network.sent‚ 6.242321e+06 disk.sda.in‚ 0.47265625 disk.rbd0.out‚0 disk.sda2.out‚0Y&Âå‚(:’ šëÒËÄÛÈ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚283.03' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.60599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_IÍêæ‚(:ß šëÒǾÊÉ' proc.memory.availableMB‚ 2406.109375 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚36811.8896484375 disk.sda1.in‚0 disk.rbd1.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚36811.8896484375 disk.sda1.out‚0 cpu‚25.5362552817828 disk.dm-0.in‚ 7197.20703125 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2872.30078125 network.sent‚ 6.247334e+06 disk.sda.out‚ 203.94921875# disk.nvme0c0n1.in‚ 7197.20703125 disk.sda3.out‚ 203.94921875* proc.memory.percent‚17.531132698059082 network.recv‚ 1.459716e+06 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 203.94921875! disk.nvme0n1.in‚ 7197.20703125$ disk./.usageGB‚585.0959281921387" disk.dm-0.out‚36811.8896484375 disk.sda3.in‚ 0.47265625$ memory_percent‚85.31427383422852) disk./.usagePercent‚39.26627866915733 disk.rbd1.out‚ 15.5390625Ì}tï2ç- _wandbruntime‚5802  _runtime‚5802u@ ÀÉèà _runtime‚5802.230630605 _step‚268 ' lr-AdamW/pg1‚0.00045689525046774824 ' lr-AdamW/pg2‚0.00045689525046774824 trainer/global_step‚3299 % _timestamp‚1.7843356416454504e+0937XÎéÈ _runtime‚5802.230630605 _step‚268 ' lr-AdamW/pg1‚0.00045689525046774824 ' lr-AdamW/pg2‚0.00045689525046774824 trainer/global_step‚3299 % _timestamp‚1.7843356416454504e+09ŒN‰8‘2ê- _wandbruntime‚5802  _runtime‚5802ÎÓqï'ëéO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28140219 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9382243 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28567798 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17849953 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8467443 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚260285763231744 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚951217995930112 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260286165859328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3097577296146 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚951216624367104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚951216624367104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚256281794678784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3104232839680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3102887450112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56717990 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28140213 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28140219 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28140213 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3102887450112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4003968552960 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _runtime‚5802.231951388 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4004003089408 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3098921896468 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17849953 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8467443 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚260285763231744 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚951217995930112 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260286165859328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3097577296146 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚951216624367104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚951216624367104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚256281794678784eC2™2í- _wandbruntime‚5803  _runtime‚5803¿O¦Aù&îóM WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4005161834496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚951081380414602 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚948391737647104 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3098500762392 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚951080010386200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚951496921702912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3105157882368 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3103812485632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28148425 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56724695 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56734688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28148419 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3099845362826 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56734688 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28586263 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚951495550132736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _runtime‚5803.642224071 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _step‚270 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚948391737647104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56724695 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚256359281086464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚947981535051776 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8469991 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28576276 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9384994 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3103812485632 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚948391763820544 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28148419 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28576276 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚256359649177600 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚948391763820544 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8469986 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚951496921702912 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17855228 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3105157882368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17854980 % _timestamp‚1.7843356430568607e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28586263 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚947981509623808†Ž;^þ&ïøM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28148425 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4005161834496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚951081380414602 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚948391737647104 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3098500762392 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚951080010386200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚951496921702912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3105157882368 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3103812485632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28148425 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56724695 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56734688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28148419 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3099845362826 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56734688 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28586263 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚951495550132736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _runtime‚5803.642224071 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _step‚270 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚948391737647104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56724695 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚256359281086464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚947981535051776 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8469991 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28576276 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9384994 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3103812485632 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚948391763820544 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28148419 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28576276 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚256359649177600 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚948391763820544 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8469986 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚951496921702912 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17855228 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3105157882368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17854980 % _timestamp‚1.7843356430568607e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28586263 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚947981509623808 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 Fa§L2‚- _wandbruntime‚5886  _runtime‚5886(æê#ï'ƒéO _step‚273 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚264219577592320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚962013799596032 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚962013773422592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚965164348366336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18116316 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚965164348366336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9521723 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9522005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚3349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚962013799596032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚964741531163686 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚965162976803328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28550513 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18116603 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚965162976803328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28550513 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18116603 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚962013799596032 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚962013773422592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚965164348366336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18116316 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚965164348366336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9521723 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9522005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚3349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚962013799596032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚964741531163686 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚965020336408108 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29002028 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8597136 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚264298656409088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚962291800326144 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3150128416256 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚962291774152704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29012015 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _runtime‚5887.499035099 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3146082475934 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28558725 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260233601470464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57570740 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚962291774152704 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57570740 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 % _timestamp‚1.7843357269136941e+09 _step‚274 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29012015 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3150128416256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4064686847488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18121651 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8597141 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚965021706436510 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57560747 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9524784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚53þ¥H0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260233969561600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28558725 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚962291800326144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9524515 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18121925 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚264298251691520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚965443274139136 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28558719 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚965441902568960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚3349 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57560747 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29002028 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3151473812992 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29002028 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3151473812992 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚965020336408108 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29002028 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8597136 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚264298656409088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚962291800326144 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3150128416256 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚962291774152704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29012015 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _runtime‚5887.499035099 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3146082475934 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28558725 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260233601470464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57570740 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚962291774152704 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57570740 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 % _timestamp‚1.7843357269136941e+09 _step‚274 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29012015 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3150128416256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4064686847488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18121651 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8597141 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚965021706436510 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57560747 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9524784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260233969561600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28558725 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚962291800326144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9524515 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820’—_á:2ˆ- _wandbruntime‚5887  _runtime‚5887r[LŠ‰ÿ 1loss/FBt2/assignment_loss‚0.00080491914051858 /loss/FRt2/detection_loss‚1.7218641502169505 %loss/total_loss‚8.574434172946749 _runtime‚5887.580874321 0loss/FRt2/assignment_loss‚0.8521989943957565 /loss/FRt1/detection_loss‚0.5698785219012467 .loss/FBt1/detection_loss‚0.511534618411501 epoch‚8 0loss/FRt1/assignment_loss‚1.7298801447842058 1loss/SRqqt1/assignment_loss‚0.591163412254291 1loss/SRqqt1/detection_loss‚0.7201385256353645 1loss/SRqqt2/detection_loss‚0.6548800727760872 trainer/global_step‚3349 $ _timestamp‚1.784335726995804e+09 _step‚275 2loss/SRqqt2/assignment_loss‚0.5232400553343577 2loss/FBt1/assignment_loss‚0.001463534132975953 /loss/FBt2/detection_loss‚0.6973872239634927s; Š„ 0loss/FRt1/assignment_loss‚1.7298801447842058 1loss/SRqqt1/assignment_loss‚0.591163412254291 1loss/SRqqt1/detection_loss‚0.7201385256353645 1loss/SRqqt2/detection_loss‚0.6548800727760872 trainer/global_step‚3349 $ _timestamp‚1.784335726995804e+09 _step‚275 2loss/SRqqt2/assignment_loss‚0.5232400553343577 2loss/FBt1/assignment_loss‚0.001463534132975953 /loss/FBt2/detection_loss‚0.6973872239634927 1loss/FBt2/assignment_loss‚0.00080491914051858 /loss/FRt2/detection_loss‚1.7218641502169505 %loss/total_loss‚8.574434172946749 _runtime‚5887.580874321 0loss/FRt2/assignment_loss‚0.8521989943957565 /loss/FRt1/detection_loss‚0.5698785219012467 .loss/FBt1/detection_loss‚0.511534618411501 epoch‚8“>ç‹‚(:’ öšëÒæ ãÉ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚251.622' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.324400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a öÕŒ‚(:Ê öšëÒÆË( proc.memory.availableMB‚ 1433.8203125 disk.drbd2165.in‚0$ disk./.usageGB‚585.0964050292969 disk.rbd1.out‚ 15.8203125% disk.nvme0c0n1.out‚36864.28515625* proc.memory.percent‚17.537283897399902 disk.sda.out‚ 207.91796875 disk.nvme0c0n1.in‚7421.75 disk.drbd2165.out‚0 network.sent‚ 6.353491e+06) disk./.usagePercent‚39.26631067009911 disk.dm-1.out‚ 207.91796875# disk.nvme0n1.out‚36864.28515625 disk.sda1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0$ memory_percent‚91.24865531921387 disk.sda3.out‚ 207.91796875 disk.sda.in‚ 0.47265625 network.recv‚ 1.484952e+06 disk.sda2.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2873.30859375 disk.dm-1.in‚ 0.47265625 disk.dm-0.out‚36864.28515625 disk.rbd1.in‚0 disk.dm-0.in‚7421.75 disk.sda3.in‚ 0.47265625 cpu‚25.535007348788106 disk.sda2.out‚0 disk.nvme0n1.in‚7421.75AK⤒‚(:‡ …›ëÒË°«Ç gpu.0.gpu‚8.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@q.­ÜŽ‚(:Ñ …›ëÒÿδÉ disk.nvme0c0n1.in‚ 7422.1875' disk.nvme0c0n1.out‚36876.5556640625' proc.memory.availableMB‚ 1433.859375$ disk./.usageGB‚585.0975532531738 disk.drbd2165.out‚0 disk.sda3.in‚ 0.47265625 cpu‚25.534639232237943 network.sent‚ 6.403421e+06 disk.rbd0.out‚0! proc.memory.rssMB‚ 2873.328125* proc.memory.percent‚17.537403106689453 disk.dm-1.out‚ 209.1015625 disk.rbd1.out‚ 15.8203125 disk.sda.in‚ 0.47265625% disk.nvme0n1.out‚36876.5556640625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625 disk.sda.out‚ 209.1015625 disk.dm-0.in‚ 7422.1875 disk.sda1.out‚0$ memory_percent‚91.24841690063477 disk.rbd1.in‚0 disk.drbd2165.in‚0" disk.dm-0.out‚36876.5556640625 disk.nvme0n1.in‚ 7422.1875 disk.sda3.out‚ 209.1015625 proc.cpu.threads‚17* disk./.usagePercent‚39.266387728366915 disk.sda1.in‚0 network.recv‚ 1.489887e+06 disk.rbd0.in‚0 \]È’‚(:‡ ”›ëÒ¿û£È gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Whiœë‚(:à ”›ëÒ™ž´É disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.in‚ 0.47265625 disk.sda.out‚ 210.2734375) disk./.usagePercent‚39.26641588919568 disk.rbd0.out‚0 network.recv‚ 1.493297e+06 disk.sda1.in‚0 cpu‚25.53417379707881( disk.nvme0c0n1.out‚36889.89697265625 disk.dm-0.in‚ 7423.15234375 disk.sda1.out‚0* proc.memory.percent‚17.539644241333008# disk./.usageGB‚585.097972869873# disk.nvme0c0n1.in‚ 7423.15234375! disk.nvme0n1.in‚ 7423.15234375& disk.nvme0n1.out‚36889.89697265625 disk.sda3.out‚ 210.2734375 disk.rbd1.out‚ 15.8828125 disk.sda3.in‚ 0.47265625$ memory_percent‚91.25144481658936" proc.memory.rssMB‚ 2873.6953125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 210.2734375# disk.dm-0.out‚36889.89697265625) proc.memory.availableMB‚ 1433.36328125 network.sent‚ 6.410419e+06FŠ³>”‘‚(:‰ £›ëÒ¹Ì²È gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚334.153' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8306 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Îåâ’‚(:× £›ëÒŠêôÊ disk.drbd2165.out‚0# disk.dm-0.out‚36892.79931640625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 15.8828125" proc.memory.rssMB‚ 2873.6953125 cpu‚25.533781075553406) disk./.usagePercent‚39.26641819326348 disk.nvme0n1.in‚ 7502.671875 disk.rbd0.out‚0 disk.dm-1.out‚ 210.796875$ disk./.usageGB‚585.0980072021484& disk.nvme0n1.out‚36892.79931640625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0# memory_percent‚91.2492036819458! disk.nvme0c0n1.in‚ 7502.671875) proc.memory.availableMB‚ 1433.73046875 network.recv‚ 1.49662e+06 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.dm-1.in‚ 0.47265625 disk.drbd2165.in‚0 network.sent‚ 6.415393e+06 disk.sda3.out‚ 210.796875 disk.dm-0.in‚ 7502.671875* proc.memory.percent‚17.539644241333008( disk.nvme0c0n1.out‚36892.79931640625 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625 disk.sda.out‚ 210.796875Ñ "ž“‚(:“ ²›ëÒìŽåÇ gpu.0.gpu‚96.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚328.786' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.75720000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®›kd锂(:Þ ²›ëÒˆ’íÉ disk.dm-0.in‚ 7503.50390625) disk./.usagePercent‚39.26642152136143 disk.rbd1.out‚ 15.9609375 disk.sda1.in‚0 disk.dm-1.in‚ 0.47265625$ disk./.usageGB‚585.0980567932129 disk.sda1.out‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚36903.494140625) proc.memory.availableMB‚ 1433.23828125" proc.memory.rssMB‚ 2873.6953125 proc.cpu.threads‚17 network.sent‚ 6.418458e+06$ disk.nvme0n1.out‚36903.494140625 disk.sda3.out‚ 211.10546875 disk.rbd0.in‚0! disk.dm-0.out‚36903.494140625* proc.memory.percent‚17.539644241333008 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 211.10546875 network.recv‚ 1.500024e+06! disk.nvme0n1.in‚ 7503.50390625 disk.drbd2165.in‚0$ memory_percent‚91.25220775604248 cpu‚25.53337130707836 disk.sda2.out‚0 disk.sda.in‚ 0.47265625 disk.sda.out‚ 211.10546875 disk.sda3.in‚ 0.47265625 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 7503.50390625Á>ôy2•- _wandbruntime‚5969  _runtime‚5969öíwMÈ–Â ' lr-AdamW/pg2‚0.00045513486944328706 trainer/global_step‚3399 % _timestamp‚1.7843358090708406e+09 _runtime‚ 5969.65596885 _step‚276 ' lr-AdamW/pg1‚0.00045513486944328706Xßø.Í—Ç ' lr-AdamW/pg1‚0.00045513486944328706 ' lr-AdamW/pg2‚0.00045513486944328706 trainer/global_step‚3399 % _timestamp‚1.7843358090708406e+09 _runtime‚ 5969.65596885 _step‚276”Cʸ2˜- _wandbruntime‚5969  _runtime‚5969s‰&Tš$™èO trainer/global_step‚3399 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚979110700802560 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _runtime‚5969.657527422 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚975491805663232 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4122792742400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚979109329239552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29429275 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264030435446784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚978681857185594 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18382796 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3196864700928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18382522 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28960813 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3191396122684 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚975913836101632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8721743 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚979109329239552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9660779 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58380115 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28960813 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚975913836101632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3195519311360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4122829376000 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚975913809928192 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973M—ì>ó'šíO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚979110700802560 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4122829376000 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚975913809928192 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 trainer/global_step‚3399 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚979110700802560 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _runtime‚5969.657527422 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚975491805663232 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4122792742400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚979109329239552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29429275 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264030435446784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚978681857185594 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18382796 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3196864700928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18382522 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28960813 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3191396122684 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚975913836101632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8721743 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚979109329239552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9660779 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58380115 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28960813 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚975913836101632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3195519311360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18387826 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚978962032458418 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚978960662430016 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29437767 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚976191836831744 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264108289945600 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3192319589042 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3196444346880 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58396799 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8724291 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚5971.083320403 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28969019 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29437767 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9663804 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28969025 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚268231893640704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚975769712869376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚975769687441408 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 trainer/global_step‚3399 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29427780 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28969019 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8724286 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58406792 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚976191810658304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58396799 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚976191836831744 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9663540 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3190974988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚264107921854464 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29427780 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 $ _timestamp‚1.784335810497865e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58406792ØÎý&Ÿ÷M YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3197789743616 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚979389626575360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚976191810658304 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18387826 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚978962032458418 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚978960662430016 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29437767 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚976191836831744 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264108289945600 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3192319589042 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3196444346880 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58396799 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8724291 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚5971.083320403 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28969019 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29437767 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9663804 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28969025 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚268231893640704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚975769712869376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚975769687441408 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 trainer/global_step‚3399 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29427780 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28969019 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8724286 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58406792 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚976191810658304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58396799 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚976191836831744 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9663540 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3190974988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚264107921854464 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29427780 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 $ _timestamp‚1.784335810497865e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58406792 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚979388255005184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3197789743616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚268232298358272–Ä)/Á2 - _wandbruntime‚5971  _runtime‚5971¨€Ÿ¡ÿ 2loss/SRqqt1/assignment_loss‚0.5826160840477551 $loss/total_loss‚9.21789995173071 epoch‚8 _step‚279 0loss/FRt1/assignment_loss‚1.6.Cý™f554824312342116 /loss/FRt2/detection_loss‚1.6791157509174728 1loss/SRqqt2/detection_loss‚0.7292140633613465 0loss/FRt2/assignment_loss‚1.0002480537219536 2loss/SRqqt2/assignment_loss‚0.5767596356201706 3loss/FBt1/assignment_loss‚0.0029606330566025463 3loss/FBt2/assignment_loss‚0.0032935053683436077 /loss/FRt1/detection_loss‚0.5229479730377345 1loss/SRqqt1/detection_loss‚0.8010149758268633 trainer/global_step‚3399 # _timestamp‚1.78433581055881e+09 /loss/FBt1/detection_loss‚0.5268628257461756 -loss/FBt2/detection_loss‚1.13738401979208 _runtime‚ 5971.14405069úámm ¢„ 0loss/FRt2/assignment_loss‚1.0002480537219536 2loss/SRqqt2/assignment_loss‚0.5767596356201706 3loss/FBt1/assignment_loss‚0.0029606330566025463 3loss/FBt2/assignment_loss‚0.0032935053683436077 /loss/FRt1/detection_loss‚0.5229479730377345 1loss/SRqqt1/detection_loss‚0.8010149758268633 trainer/global_step‚3399 # _timestamp‚1.78433581055881e+09 /loss/FBt1/detection_loss‚0.5268628257461756 -loss/FBt2/detection_loss‚1.13738401979208 _runtime‚ 5971.14405069 2loss/SRqqt1/assignment_loss‚0.5826160840477551 $loss/total_loss‚9.21789995173071 epoch‚8 _step‚279 0loss/FRt1/assignment_loss‚1.6554824312342116 /loss/FRt2/detection_loss‚1.6791157509174728 1loss/SRqqt2/detection_loss‚0.7292140633613465—I☾ž£‚(:“ ЛëÒ’ì«È gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚346.118' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚69.22359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®¬¡ñ¤‚(:æ ЛëÒ˜ÖúÉ) disk./.usagePercent‚39.27489204264684 proc.cpu.threads‚17 disk.sda3.out‚ 216.42578125 disk.sda.out‚ 216.42578125* proc.memory.percent‚17.540478706359863 network.sent‚ 6.475687e+06 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 16.0234375 disk.rbd0.in‚0 disk.sda.in‚ 0.47265625# disk.nvme0c0n1.in‚ 7504.55078125# proc.memory.rssMB‚ 2873.83203125 network.recv‚ 1.507726e+06! disk.nvme0n1.in‚ 7504.55078125 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625$ memory_percent‚91.25220775604248 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.dm-0.out‚37063.05712890625 cpu‚25.532579871519566$ disk./.usageGB‚585.2242736816406( disk.nvme0c0n1.out‚37063.05712890625) proc.memory.availableMB‚ 1433.23828125& disk.nvme0n1.out‚37063.05712890625 disk.dm-1.out‚ 216.42578125 disk.dm-0.in‚ 7504.55078125V„÷”¥‚(:‰ ß›ëÒÊÈÇ gpu.0.gpu‚82.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚324.933' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.9866 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:¡gÙ¦‚(:Î ß›ëÒ¯ÀôÈ network.recv‚ 1.512565e+06# disk.nvme0c0n1.out‚ 37071.234375 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 217.046875 disk.dm-0.out‚ 37071.234375 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 7585.37109375 disk.sda3.in‚ 0.47265625 disk.rbd0.out‚0 disk.sda.out‚ 217.046875( proc.memory.availableMB‚ 1433.0703125 cpu‚25.5321761219121 network.sent‚ 6.528015e+06 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 7585.37109375 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚39.27491738739273 disk.sda2.out‚0$ memory_percent‚91.25323295593262* proc.memory.percent‚17.540669441223145 proc.cpu.threads‚17 disk.rbd1.out‚ 16.0234375 disk.sda.in‚ 0.47265625 disk.sda3.out‚ 217.046875$ disk./.usageGB‚585.2246513366699! disk.nvme0n1.in‚ 7585.37109375! disk.nvme0n1.out‚ 37071.234375# proc.memory.rssMB‚ 2873.86328125 Z±¥’§‚(:‡ î›ëÒü¦¶È gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚334.442' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È™uþ騂(:Þ î›ëÒ½¡ÆÉ disk.dm-0.in‚ 7586.3359375 disk.sda3.out‚ 217.5703125 disk.rbd0.out‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 7586.3359375$ disk./.usageGB‚585.2246856689453 disk.nvme0n1.in‚ 7586.3359375 disk.rbd1.in‚0( disk.nvme0c0n1.out‚37083.55029296875) proc.memory.availableMB‚ 1432.81640625 disk.sda1.out‚0 disk.rbd1.out‚ 16.1015625 disk.drbd2165.in‚0$ memory_percent‚91.25478267669678 network.sent‚ 6.53305e+06 network.recv‚ 1.516178e+06 proc.cpu.threads‚17 disk.sda.out‚ 217.5703125# proc.memory.rssMB‚ 2873.88671875* proc.memory.percent‚17.540812492370605 cpu‚25.53180083253544) disk./.usagePercent‚39.27491969146054 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 217.5703125 disk.sda3.in‚ 0.47265625# disk.dm-0.out‚37083.55029296875& disk.nvme0n1.out‚37083.55029296875 disk.sda.in‚ 0.47265625í3„!”©‚(:‰ ý›ëÒÄ°«Ç gpu.0.gpu‚44.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.952' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„Ì* ت‚(:Í ý›ëÒ¯ˆÿÈ disk.drbd2165.out‚0 disk.nvme0n1.in‚7586.375$ disk.nvme0n1.out‚37102.416015625 disk.sda1.in‚0* proc.memory.percent‚17.540955543518066 proc.cpu.threads‚17! disk.dm-0.out‚37102.416015625 disk.sda1.out‚0 disk.sda3.out‚ 219.30078125 disk.sda2.out‚0# proc.memory.rssMB‚ 2873.91015625 cpu‚25.531272814253416 network.sent‚ 6.53801e+06 disk.dm-1.out‚ 219.30078125* disk./.usagePercent‚39.275410457903675 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625 disk.rbd0.out‚0$ memory_percent‚91.25480651855469 disk.sda3.in‚ 0.47265625 disk.dm-1.in‚ 0.47265625 network.recv‚ 1.519442e+06 disk.rbd1.out‚ 16.1015625 disk.nvme0c0n1.in‚7586.375& disk.nvme0c0n1.out‚37102.416015625 disk.dm-0.in‚7586.375$ disk./.usageGB‚585.2319984436035 disk.sda.out‚ 219.30078125 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0% proc.memory.availableMB‚ 1432.8125êœl«‚(:’ ŒœëÒ±–¢Ç gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.164599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00j!í¬‚(:â ŒœëÒ¡®É disk.drbd2165.in‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 2873.9140625 disk.sda.out‚ 222.57421875" disk.nvme0c0n1.in‚ 7587.6015625( disk.nvme0c0n1.out‚37117.14404296875 disk.sda2.out‚0$ memory_percent‚91.25478267669678 cpu‚25.53094518016575 proc.cpu.threads‚17 disk.rbd1.in‚0# disk.dm-0.out‚37117.14404296875 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 222.57421875 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0 disk.dm-0.in‚ 7587.6015625 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625) proc.memory.availableMB‚ 1432.81640625 network.recv‚ 1.522781e+06* proc.memory.percent‚17.540979385375977$ disk./.usageGB‚585.2313842773438 disk.sda3.out‚ 222.57421875 disk.sda2.in‚0 network.sent‚ 6.543082e+06* disk./.usagePercent‚39.275369240690665 disk.rbd1.out‚ 16.2265625 disk.nvme0n1.in‚ 7587.6015625& disk.nvme0n1.out‚37117.14404296875 disk.sda1.out‚0 disk.rbd0.out‚0…F»ˆ2­- _wandbruntime‚6053  _runtime‚6053íÝVÇ®Á & lr-AdamW/pg1‚0.0004542340430232862 & lr-AdamW/pg2‚0.0004542340430232862 trainer/global_step‚3449 % _timestamp‚1.7843358929236324e+09 _runtime‚6053.508856766 _step‚2808´þ€Ì¯Æ & lr-AdamW/pg2‚0.0004542340430232862 trainer/global_step‚3449 % _timestamp‚1.7843358929236324e+09 _runtime‚6053.508856766 _step‚280 & lr-AdamW/pg1‚0.0004542340430232862˜znHn2°- _wandbruntime‚6053  _runtime‚6053|f‡ùï'±éO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29371119 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3243180631552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9799846 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3241835241984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4182235224576 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29371113 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚267905123921920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚993057053238784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚989813872607232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚993057053238784 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚3449 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚992623553235792 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚989813846433792 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3241835241984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4182269761024 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚993055681675776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3236288635470 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29845054 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚993055681675776 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29855027 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3243180631552 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0òIBºô'²îO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59226146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29371119 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3243180631552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9799846 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3241835241984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4182235224576 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29371113 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚267905123921920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚993057053238784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚989813872607232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚993057053238784 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚3449 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚992623553235792 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚989813846433792 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3241835241984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4182269761024 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚993055681675776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3236288635470 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29845054 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚993055681675776 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29855027 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3243180631552 I*ñ[2³- _wandbruntime‚6054  _runtime‚6054^ŠÞ‹ ´òM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29853532 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 trainer/global_step‚3449 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚990091847163904 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚992902358480326 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚993335979011584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚990091847163904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29379319 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29863519 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4183457235456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29379325 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚267982610329600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3242760277504 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272165664944640 HBDeviceStatsMonitor.on_train_batch_end/ov¿ÉÜÿmersize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚282 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18654010 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272166067565056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29863519 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18653761 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8851436 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59242844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3244105674240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3238556702150 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3237212101716 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚990091873337344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚993335979011584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29379325 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9802569 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚989663801778176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 trainer/global_step‚3449 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚990091847163904 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚992902358480326 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚993335979011584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚990091847163904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29379319 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29863519 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4183457235456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29379325 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚267982610329600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3242760277504 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272165664944640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚282 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18654010 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272166067565056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29863519 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18653761 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8851436 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59242844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3244105674240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3238556702150 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3237212101716 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚990091873337344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚993335979011584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29379325 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9802569 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚989663801778176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1003279983480832 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29781419 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18914873 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚271779444305920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3289496562176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276020990971392 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1007003405675008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18914617 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1007002034112000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30280779 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4241949293056 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30280779 VFDeviceStatsMonitor.on_train_¾J­1#batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8976048 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3283870348900 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1006562509229410 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3282525748578 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3288151172608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30270806 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1003713882939392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4241914756608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1007002034112000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30280779 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4241949293056 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30280779 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8976048 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3283870348900 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1006562509229410 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3282525748578 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3288151172608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30270806 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1003713882939392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4241914756608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1003279983480832 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29781419 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18914873 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚271779444305920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3289496562176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276020990971392 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1007003405675008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18914617 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784335977969232e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30289271 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3290421604864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18920188 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1003557865259008 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3290421604864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30279284 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30289271 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9941597 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1007282331447808 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29789619 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60078896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1006842684502234 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1003991909842944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60068903 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29789625 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚286 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29789619 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6138.554628349 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18919934 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8978586 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚271856930713600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30279284 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1003991909842944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 trainer/global_step‚3499 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8978591 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3289076208128 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1007280959877632 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚271856562622464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3284793815258 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3283449214824 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60078896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1006842684502234 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1003991909842944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60068903 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29789625 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚286 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29789619 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6138.554628349 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18919934 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8978586 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚271856930713600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30279284 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1003991909842944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 trainer/global_step‚3499 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8978591 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3289076208128 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1007280959877632 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚271856562622464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3284793815258 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3283449214824 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784335977969232e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30289271 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3290421604864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18920188 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freM"Ž… ed‚1003557865259008 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3290421604864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30279284 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30289271 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9941597 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1007282331447808 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29789619 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ž:m42Î- _wandbruntime‚6138  _runtime‚613813ú@Ïþ 2loss/FBt1/assignment_loss‚0.013045210602959045 1loss/FBt2/assignment_loss‚0.00953037034189453 /loss/FRt1/detection_loss‚0.5423325793238908 /loss/FRt2/detection_loss‚1.7515073247823885 1loss/SRqqt1/detection_loss‚0.7606065819737227 0loss/FRt2/assignment_loss‚0.8472328342375208 $loss/total_loss‚9.08267937890213 epoch‚8 0loss/FRt1/assignment_loss‚1.7913616381271589 2loss/SRqqt2/assignment_loss‚0.5200758205066426 0loss/SRqqt2/detection_loss‚0.691195969861509 .loss/FBt1/detection_loss‚0.507894223299124 _runtime‚ 6138.60039081 _step‚287 2loss/SRqqt1/assignment_loss‚0.5405861272849761 /loss/FBt2/detection_loss‚1.1073106985603427 trainer/global_step‚3499 % _timestamp‚1.7843359780153103e+09m0÷B Ѓ 0loss/FRt2/assignment_loss‚0.8472328342375208 $loss/total_loss‚9.08267937890213 epoch‚8 0loss/FRt1/assignment_loss‚1.7913616381271589 2loss/SRqqt2/assignment_loss‚0.5200758205066426 0loss/SRqqt2/detection_loss‚0.691195969861509 .loss/FBt1/detection_loss‚0.507894223299124 _runtime‚ 6138.60039081 _step‚287 2loss/SRqqt1/assignment_loss‚0.5405861272849761 /loss/FBt2/detection_loss‚1.1073106985603427 trainer/global_step‚3499 % _timestamp‚1.7843359780153103e+09 2loss/FBt1/assignment_loss‚0.013045210602959045 1loss/FBt2/assignment_loss‚0.00953037034189453 /loss/FRt1/detection_loss‚0.5423325793238908 /loss/FRt2/detection_loss‚1.7515073247823885 1loss/SRqqt1/detection_loss‚0.7606065819737227Ÿ®ý¿¬’Ñ‚(:‡ õœëÒ„½´Ç gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç™©ÐîÒ‚(:ã õœëÒíåªÉ disk.drbd2165.in‚0 disk.sda1.in‚0 cpu‚25.528192588289347 disk.sda.in‚ 0.47265625! disk.nvme0n1.in‚ 7773.36328125 disk.sda3.out‚ 237.29296875 disk.dm-1.in‚ 0.47265625* proc.memory.percent‚17.541217803955078! proc.memory.rssMB‚ 2873.953125 network.sent‚ 6.635159e+06 network.recv‚ 1.546823e+06 disk.rbd1.in‚0 disk.rbd1.out‚ 16.4453125 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 237.29296875# disk.nvme0c0n1.in‚ 7773.36328125# disk.dm-0.out‚38476.33740234375& disk.nvme0n1.out‚38476.33740234375 disk.dm-1.out‚ 237.29296875) disk./.usagePercent‚39.35713318298386 disk.dm-0.in‚ 7773.36328125( proc.memory.availableMB‚ 1432.4453125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚91.25704765319824$ disk./.usageGB‚586.4497261047363( disk.nvme0c0n1.out‚38476.33740234375 disk.sda3.in‚ 0.47265625 proc.cpu.threads‚17i,ÞñÓ‚(:’ „ëÒ’¶ÄÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚281.47' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.29400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü-;këÔ‚(:à „ëÒ÷­ÁÊ disk.dm-1.in‚ 0.47265625 disk.sda3.out‚ 238.58203125$ memory_percent‚91.25707149505615 disk.rbd1.out‚ 16.5078125$ disk./.usageGB‚586.4555320739746 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 7774.328125& disk.nvme0n1.out‚38509.46044921875# proc.memory.rssMB‚ 2873.96484375 proc.cpu.threads‚17 network.sent‚ 6.698385e+06 disk.sda2.in‚0( disk.nvme0c0n1.out‚38509.46044921875 disk.dm-0.in‚ 7774.328125 disk.rbd0.out‚0 disk.sda.in‚ 0.47265625 disk.sda.out‚ 238.58203125 disk.sda1.in‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1432.44140625 disk.sda2.out‚0 disk.dm-1.out‚ 238.58203125 cpu‚25.527831488168115* disk./.usagePercent‚39.357522826450975 disk.drbd2165.out‚0# disk.dm-0.out‚38509.46044921875 disk.sda1.out‚0 network.recv‚ 1.551525e+06! disk.nvme0c0n1.in‚ 7774.328125 disk.sda3.in‚ 0.47265625) proc.memory.percent‚17.54128932952881…bñÒ”Õ‚(:‰ “ëÒžÊæÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚266.957' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝêwçéÖ‚(:Þ “ëÒƒ™¼É! disk.nvme0n1.in‚ 7846.45703125 network.sent‚ 6.703463e+06$ disk./.usageGB‚586.4501609802246# disk.nvme0c0n1.in‚ 7846.45703125& disk.nvme0n1.out‚38522.44482421875 disk.sda1.in‚0 disk.rbd1.in‚0) proc.memory.percent‚17.54140853881836 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0$ memory_percent‚91.25709533691406 cpu‚25.527450793368754) disk./.usagePercent‚39.35716236784277 disk.sda.out‚ 240.60546875 disk.drbd2165.in‚0 disk.dm-0.in‚ 7846.45703125# disk.dm-0.out‚38522.44482421875 disk.rbd1.out‚ 16.5078125! proc.memory.rssMB‚ 2873.984375 disk.sda.in‚ 0.47265625 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 240.60546875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 240.60546875% proc.memory.availableMB‚ 1432.4375 proc.cpu.threads‚17 disk.dm-1.in‚ 0.47265625 network.recv‚ 1.55503e+06( disk.nvme0c0n1.out‚38522.44482421875–ï¨!Ÿ×‚(:” ¢ëÒ‡òùÇ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚273.071' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.614200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-[®tæØ‚(:Û ¢ëÒì»ÒÉ& disk.nvme0c0n1.out‚38540.650390625 disk.sda2.in‚0) disk./.usagePercent‚39.35756353164892 disk.sda.in‚ 0.47265625 disk.sda3.out‚ 242.86328125 disk.rbd0.out‚0! disk.dm-0.out‚38540.650390625 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0! disk.nvme0n1.in‚ 7846.45703125 disk.rbd0.in‚0% proc.memory.availableMB‚ 1432.5625 proc.cpu.threads‚17 disk.rbd1.out‚ 16.6484375 disk.drbd2165.in‚0 disk.sda3.in‚ 0.47265625 disk.dm-1.in‚ 0.47265625) proc.memory.percent‚17.54143238067627 cpu‚25.527061563739675$ disk.nvme0n1.out‚38540.650390625 disk.sda2.out‚0$ memory_percent‚91.25633239746094# proc.memory.rssMB‚ 2873.98828125 disk.dm-0.in‚ 7846.45703125 network.recv‚ 1.558187e+06# disk.nvme0c0n1.in‚ 7846.45703125 disk.sda1.in‚0 network.sent‚ 6.708602e+06 disk.dm-1.out‚ 242.86328125$ disk./.usageGB‚586.4561386108398 disk.sda.out‚ 242.86328125€ªß”Ù‚(:‰ ±ëÒ‹¼»È gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚296.751' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.3502 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™ôÏèåÚ‚(:Ú ±ëÒêöæÉ disk.sda2.in‚0 disk.sda.out‚ 244.67578125$ disk./.usageGB‚586.4546928405762 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 7847.421875$ memory_percent‚91.25633239746094 network.recv‚ 1.56135e+06 disk.nvme0n1.in‚ 7847.421875 disk.rbd0.out‚0 disk.sda.in‚ 0.47265625! disk.nvme0c0n1.in‚ 7847.421875& disk.nvme0n1.out‚38554.36474609375 disk.sda2.out‚0 disk.rbd1.out‚ 16.6484375 disk.drbd2165.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17# disk.dm-0.out‚38554.36474609375# proc.memory.rssMB‚ 2873.98828125 cpu‚25.526874802061858) disk./.usagePercent‚39.35746650479344 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 244.67578125 disk.dm-1.out‚ 244.67578125( disk.nvme0c0n1.out‚38554.36474609375% proc.memory.availableMB‚ 1432.5625) proc.memory.percent‚17.54143238067627 network.sent‚ 6.713464e+06 disk.sda1.in‚0 †]2Û- _wandbruntime‚6220  _runtime‚62209¡µ!ÇÜÁ trainer/global_step‚3549 % _timestamp‚1.7843360601464515e+09 _runtime‚6220.731954515 _step‚288 & lr-AdamW/pg1‚0.0004523914076560161 & lr-AdamW/pg2‚0.0004523914076560161¾ ÌÝÆ & lr-AdamW/pg1‚0.0004523914076560161 & lr-AdamW/pg2‚0.0004523914076560161 trainer/global_step‚3549 % _timestamp‚1.7843360601464515e+09 _runtime‚6220.731954515 _step‚288 9-‚2Þ- _wandbruntime‚6220  _runtime‚6220Ì_œUû'ßõO VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1020948386548224 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚279955069175296 _runtime‚6220.734256922 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3335812492800 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4301707112960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3334467103232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19180848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1017613945618432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30191713 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30706531 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30706531 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1017613945618432 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1020949758111232 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9103198 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10077650 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30191713 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1017613919444992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9103193 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30696558 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1020948386548224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4301672576512 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1017174097817600 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 trainer/global_step‚3549 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10077404 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 490712596489|Í(àúO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1020948386548224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4301672576512 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1017174097817600 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 trainer/global_step‚3549 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10077404 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1020948386548224 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚279955069175296 _runtime‚6220.734256922 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3335812492800 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4301707112960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3334467103232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19180848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1017613945618432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30191713 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30706531 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30706531 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1017613945618432 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1020949758111232 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9103198 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10077650 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30191713 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1017613919444992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9103193 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30696558¡‚àÑo2á- _wandbruntime‚6222  _runtime‚6222mjø&ûâ€N HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚280034154024448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _JDeviceStat–ó  %sMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1017891946348544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1021228683884032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275731251097600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4302902926848 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30705036 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3335392138752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30199925 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _step‚290 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19185879 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60914948 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10080143 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3336737535488 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4302868397568 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1020783010524142 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1017891920175104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3329686327932 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30715023 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1017891920175104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30199919 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30199925 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30715023 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1020781640495740 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784336061559738e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9105736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30199919 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3336737535488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9105741 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10080387 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3335392138752 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1017451954167808 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60914948 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1e÷½é 'ã…N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30199919 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3336737535488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9105741 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10080387 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3335392138752 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1017451954167808 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60914948 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚280034154024448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1017891946348544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1021228683884032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275731251097600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4302902926848 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30705036 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3335392138752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30199925 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _step‚290 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19185879 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60914948 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10080143 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3336737535488 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4302868397568 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1020783010524142 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1017891920175104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3329686327932 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30715023 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1017891920175104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30199919 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30199925 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30715023 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1020781640495740 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784336061559738e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9105736¢yz°2ä- _wandbruntime‚6222  _runtime‚6222¥»¥zå /loss/FRt2/detection_loss‚1.5677256088649087 epoch‚8 _step‚291 .loss/FBt2/detection_loss‚0.875246199878997 2loss/SRqqt1/assignment_loss‚0.5962136704621702 2loss/SRqqt2/assignment_loss‚0.5621915072083339 0loss/SRqqt2/detection_loss‚0.742467547632567 % _timestamp‚1.7843360616368804e+09 %loss/total_loss‚8.923231327603233 0loss/FRt2/assignment_loss‚0.8793877012589398 1loss/FBt1/assignment_loss‚0.05041265649490439 /loss/FRt1/detection_loss‚0.5573754027838377 1loss/SRqqt1/detection_loss‚0.7701587273976432 /loss/FBt1/detection_loss‚0.5904773260711712 trainer/global_step‚3549 _runtime‚6222.221987606 0loss/FRt1/assignment_loss‚1.7278091232709967 3loss/FBt2/assignment_loss‚0.0037658562787617556Ã—Æ æ† 1loss/SRqqt1/detection_loss‚0.7701587273976432 /loss/FBt1/detection_loss‚0.5904773260711712 trainer/global_step‚3549 _runtime‚6222.221987606 0loss/FRt1/assignment_loss‚1.7278091232709967 3loss/FBt2/assignment_loss‚0.0037658562787617556 /loss/FRt2/detection_loss‚1.5677256088649087 epoch‚8 _step‚291 .loss/FBt2/detection_loss‚0.875246199878997 2loss/SRqqt1/assignment_loss‚0.5962136704621702 2loss/SRqqt2/assignment_loss‚0.5621915072083339 0loss/SRqqt2/detection_loss‚0.742467547632567 % _timestamp‚1.7843360616368804e+09 %loss/total_loss‚8.923231327603233 0loss/FRt2/assignment_loss‚0.8793877012589398 1loss/FBt1/assignment_loss‚0.05041265649490439 /loss/FRt1/detection_loss‚0.5573754027838377£ÓnÅ”ç‚(:‰ ÀëÒ„©ÕÈ gpu.0.gpu‚84.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.219' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.4438 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù{ðè‚(:å ÀëÒ†ý‡Ê disk.rbd1.out‚ 16.7109375( disk.nvme0c0n1.out‚38568.43505859375 disk.sda1.out‚0# disk.dm-0.out‚38568.43505859375 network.recv‚ 1.564328e+06 disk.sda3.out‚ 244.99609375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 7848.25390625) proc.memory.availableMB‚ 1432.33203125 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 7848.25390625 disk.sda.in‚ 0.47265625 disk.sda.out‚ 244.99609375! disk.nvme0n1.in‚ 7848.25390625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.out‚ 244.99609375& disk.nvme0n1.out‚38568.43505859375$ memory_percent‚91.25773906707764# proc.memory.rssMB‚ 2873.98828125 cpu‚25.526448101608754 network.sent‚ 6.718445e+06) disk./.usagePercent‚39.35746880886125 disk.sda1.in‚0 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0$ disk./.usageGB‚586.4547271728516 disk.sda3.in‚ 0.47265625) proc.memory.percent‚17.54143238067627÷ÒݵŸé‚(:” ÏëÒØËÑÇ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.062' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.812400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YLÍæê‚(:Û ÏëÒ±Ý’Ë disk.drbd2165.in‚0 disk.sda3.in‚ 0.47265625$ memory_percent‚91.25723838806152# proc.memory.rssMB‚ 2873.98828125 network.sent‚ 6.781682e+06* disk./.usagePercent‚39.357242754208514 disk.sda2.in‚0 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 7948.203125" disk.dm-0.out‚38572.3642578125 disk.nvme0n1.in‚ 7948.203125 disk.rbd0.in‚0 disk.dm-1.out‚ 245.38671875 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625# disk./.usageGB‚586.451358795166 disk.drbd2165.out‚0 disk.sda1.in‚0) proc.memory.percent‚17.54143238067627 cpu‚25.526096414548448 disk.rbd0.out‚0' disk.nvme0c0n1.out‚38572.3642578125 network.recv‚ 1.569154e+06 disk.rbd1.in‚0 disk.sda.out‚ 245.38671875 disk.dm-0.in‚ 7948.203125% disk.nvme0n1.out‚38572.3642578125 disk.sda3.out‚ 245.38671875( proc.memory.availableMB‚ 1432.4140625 disk.rbd1.out‚ 16.7109375 disk.sda.in‚ 0.47265625Ò’\žë‚(:“ ÞëÒðÏðÈ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚329.465' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.89299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚÔÍ ãì‚(:Ø ÞëÒнÌÉ# proc.memory.rssMB‚ 2873.98828125 disk.sda.out‚ 247.78125 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 7949.16796875 disk.sda3.out‚ 247.78125 disk.dm-1.in‚ 0.47265625 disk.rbd1.out‚ 16.7890625) proc.memory.percent‚17.54143238067627% disk.nvme0n1.out‚38587.4970703125 disk.sda2.out‚0$ disk./.usageGB‚586.4482879638672 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.sda.in‚ 0.47265625! disk.nvme0n1.in‚ 7949.16796875 disk.rbd1.in‚0) proc.memory.availableMB‚ 1432.40234375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚38587.4970703125" disk.dm-0.out‚38587.4970703125 disk.sda2.in‚0$ memory_percent‚91.25730991363525 cpu‚25.52568881499981 network.sent‚ 6.786664e+06 network.recv‚ 1.572409e+06) disk./.usagePercent‚39.35703666814345 proc.cpu.threads‚17 disk.dm-0.in‚ 7949.16796875 disk.sda1.in‚0 disk.dm-1.out‚ 247.78125ÇûÍm”í‚(:‰ íëÒýú¸È gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚237.057' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.4114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 §ìî‚(:á íëÒÔÉšÉ disk.rbd0.in‚0 disk.dm-1.in‚ 0.47265625 disk.drbd2165.out‚0 disk.dm-0.in‚ 7949.16796875$ memory_percent‚91.25726222991943 disk.sda1.in‚0 disk.sda3.out‚ 250.21484375# disk./.usageGB‚586.449291229248! disk.nvme0n1.in‚ 7949.16796875 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625" disk.dm-0.out‚38596.3349609375 disk.sda2.out‚0) proc.memory.percent‚17.54143238067627 cpu‚25.525313203349103) proc.memory.availableMB‚ 1432.41015625 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 7949.16796875' disk.nvme0c0n1.out‚38596.3349609375 network.recv‚ 1.575946e+06) disk./.usagePercent‚39.35710399812496 disk.sda2.in‚0 disk.dm-1.out‚ 250.21484375 disk.sda.in‚ 0.47265625 network.sent‚ 6.791532e+06% disk.nvme0n1.out‚38596.3349609375 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 250.21484375 proc.cpu.threads‚17 disk.rbd1.out‚ 16.7890625# proc.memory.rssMB‚ 2873.98828125$–¯Ÿï‚(:” üëÒ–Ù·Ç gpu.0.gpu‚95.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚250.801' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.160199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g&EÇð‚(:¼ üëÒúеÉ disk.sda2.in‚0 disk.sda1.out‚0' disk./.usagePercent‚39.357078141364 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚7950 disk.sda3.out‚ 251.65625 disk.rbd0.out‚0' proc.memory.availableMB‚ 1432.140625 network.sent‚ 6.796553e+06 disk.rbd1.out‚ 16.8515625 disk.dm-0.in‚7950 disk.sda3.in‚ 0.47265625 disk.sda.in‚ 0.47265625( disk.nvme0c0n1.out‚38608.17626953125) proc.memory.percent‚17.54152774810791 proc.cpu.threads‚17 disk.nvme0n1.in‚7950& disk.nvme0n1.out‚38608.17626953125$ memory_percent‚91.25890731811523# proc.memory.rssMB‚ 2874.00390625$ disk./.usageGB‚586.4489059448242 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 cpu‚25.52497391162845 network.recv‚ 1.579315e+06 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 251.65625 disk.sda.out‚ 251.65625# disk.dm-0.out‚38608.17626953125 disk.sda1.in‚0`?%9žñ‚(:“ ‹žëÒÆíÀÇ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚294.335' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.86699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 {Gáò‚(:Ö ‹žëÒ¹ÑÈÉ disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0 disk.sda3.out‚ 252.12109375 disk.rbd0.out‚0' proc.memory.availableMB‚ 1432.140625) disk./.usagePercent‚39.35707865337907 disk.dm-1.out‚ 252.12109375$ memory_percent‚91.25890731811523 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 8035.39453125 disk.dm-0.out‚ 38617.3828125 disk.dm-1.in‚ 0.47265625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 network.sent‚ 6.801632e+06 network.recv‚ 1.583066e+06" disk.nvme0n1.out‚ 38617.3828125 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.rbd1.in‚0$ disk.nvme0c0n1.out‚ 38617.3828125 disk.rbd1.out‚ 16.8515625 disk.sda.out‚ 252.12109375! disk.nvme0n1.in‚ 8035.39453125 disk.sda2.in‚0 cpu‚25.524597236401025$ disk./.usageGB‚586.4489135742188 disk.dm-0.in‚ 8035.39453125" proc.memory.rssMB‚ 2874.0078125) proc.memory.percent‚17.54155158996582 proc.cpu.threads‚17p1÷Ÿ2ó- _wandbruntime‚6304  _runtime‚6304°ƒÍÖÉôà trainer/global_step‚3599 % _timestamp‚1.7843361440606027e+09 _runtime‚6304.645784167 _step‚292 ' lr-AdamW/pg1‚0.00045144971705777316 ' lr-AdamW/pg2‚0.00045144971705777316~RŸLÎõÈ trainer/global_step‚3599 % _timestamp‚1.7843361440606027e+09 _runtime‚6304.645784167 _step‚292 ' lr-AdamW/pg1‚0.00045144971705777316 ' lr-AdamW/pg2‚0.00045144971705777316¤ì¨\@2ö- _wandbruntime‚6304  _runtime‚6304Hø Ÿ! ÷÷O P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30602013 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1031068161298432 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1031513982124032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3380783033856 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31122310 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚279527716982784 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1034894738984448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚283889257604608 trainer/global_step‚3599 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3380783033856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31132283 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚279528085073920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1034444531301516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1034896110547456 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1034443161273226 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1031513955950592 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4361540621824 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30602019 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19446674 _step‚293 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9230343 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30602013 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3374999974794 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10216584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648ÏÁV(øüO ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1034896110547456 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1034443161273226 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1031513955950592 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4361540621824 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30602019 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19446674 _step‚293 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9230343 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30602013 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3374999974794 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10216584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30602013 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1031068161298432 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1031513982124032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3380783033856 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31122310 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚279527716982784 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1034894738984448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚283889257604608 trainer/global_step‚3599 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3380783033856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31132283 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚279528085073920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1034444531301516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820¥©ß2ù- _wandbruntime‚6306  _runtime‚6306Ø"Å%'úN KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚6306.047992149 _step‚294 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61751000 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3381708069376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4362765581824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚3599 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1031346043076608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19451997 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3383053466112 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9232891 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚279605203390464 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1034723336546050 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283968337063424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30610219 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚283967934443008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 % _timestamp‚1.7843361454626539e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61741007 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9232886 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3375923441040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19452240 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31130788 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30610225 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30610219 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1031791956680704 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61751000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚279605571481600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1035173664750080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31130788 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4362731052544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31130788 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4362731052544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1031346043076608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19451997 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3383053466112 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9232891 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚279605203390464 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1034723336546050 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283968337063424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30610219 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚283967934443008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 % _timestamp‚1.7843361454626539e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61741007 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9232886 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3375923441040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19452240 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31130788 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30610225 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30610219 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1031791956680704 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61751000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚279605571481600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1035173664750080¦5Šº2ü- _wandbruntime‚6306  _runtime‚6306ð^$! ý„ 1loss/SRqqt1/detection_loss‚0.8136625244931889 _runtime‚6306.129437354 0loss/FRt1/assignment_loss‚1.8049119241806195 0loss/FRt2/assignment_loss‚0.8112993122002725 3loss/SRqqt2/assignment_loss‚0.48916028920932614 /loss/FRt1/detection_loss‚0.5422670302189538 %loss/total_loss‚9.011613853443876 _step‚295 2loss/SRqqt1/assignment_loss‚0.5269695226836761 2loss/FBt1/assignment_loss‚0.006707256532584738 2loss/FBt2/assignment_loss‚0.061963736522243705 /loss/FRt2/detection_loss‚1.6435566598986182 /loss/FBt2/detection_loss‚0.9428431325725146 epoch‚8 trainer/global_step‚3599 1loss/SRqqt2/detection_loss‚0.6861945420805824 /loss/FBt1/detection_loss‚0.6820779228512954 % _timestamp‚1.7843361455441847e+09ÝÍUYþ‰ /loss/FBt1/detection_loss‚0.6820779228512954 % _timestamp‚1.7843361455441847e+09 1loss/SRqqt1/detection_loss‚0.8136625244931889 _runtime‚6306.129437354 0loss/FRt1/assignment_loss‚1.8049119241806195 0loss/FRt2/assignment_loss‚0.8112993122002725 3loss/SRqqt2/assignment_loss‚0.48916028920932614 /loss/FRt1/detection_loss‚0.5422670302189538 %loss/total_loss‚9.011613853443876 _step‚295 2loss/SRqqt1/assignment_loss‚0.5269695226836761 2loss/FBt1/assignment_loss‚0.006707256532584738 2loss/FBt2/assignment_loss‚0.061963736522243705 /loss/FRt2/detection_loss‚1.6435566598986182 /loss/FBt2/detection_loss‚0.9428431325725146 epoch‚8 trainer/global_step‚3599 1loss/SRqqt2/detection_loss‚0.6861945420805824§“ª‹ÿ‚(:’ šžëÒþñ¢È gpu.0.gpu‚96.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚331.35' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.27000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œéçÛ倂(:Ú šžëÒ‹Ë„Ì# disk.nvme0c0n1.in‚ 8035.39453125 disk.sda2.in‚0 disk.rbd1.out‚ 16.9296875 disk.drbd2165.out‚0 cpu‚25.527475695109924 network.sent‚ 6.819801e+06) disk./.usagePercent‚39.35698930674962 disk.sda.out‚ 252.87890625 disk.drbd2165.in‚0$ memory_percent‚91.30334854125977' proc.memory.availableMB‚ 1424.859375 proc.cpu.threads‚17 disk.rbd1.in‚0( proc.memory.percent‚17.5872802734375( disk.nvme0c0n1.out‚38623.83935546875# disk.dm-0.out‚38623.83935546875# disk./.usageGB‚586.447582244873& disk.nvme0n1.out‚38623.83935546875 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 252.87890625 network.recv‚ 1.586337e+06! disk.nvme0n1.in‚ 8035.39453125 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 252.87890625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda.in‚ 0.47265625 disk.dm-0.in‚ 8035.39453125 proc.memory.rssMB‚2881.5F4βŸ‚(:” ©žëÒ¡›ÆÇ gpu.0.gpu‚64.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–Dìê‚‚(:ß ©žëÒô‰œÉ$ memory_percent‚91.39642715454102 disk.sda.in‚ 0.47265625" disk.nvme0c0n1.in‚ 8036.2265625& disk.nvme0n1.out‚38633.24169921875 disk.sda1.in‚0 disk.sda3.out‚ 253.19140625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.out‚ 253.19140625 disk.dm-0.in‚ 8036.2265625# disk.dm-0.out‚38633.24169921875 disk.sda1.out‚0* disk./.usagePercent‚39.356996986975645# disk./.usageGB‚586.447696685791 disk.nvme0n1.in‚ 8036.2265625 disk.sda3.in‚ 0.47265625* proc.memory.percent‚17.679309844970703 network.sent‚ 6.869757e+06 proc.cpu.threads‚17 network.recv‚ 1.591183e+06 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 253.19140625 disk.rbd1.out‚ 16.9296875 cpu‚25.533492506814284 disk.sda2.out‚0( disk.nvme0c0n1.out‚38633.24169921875 disk.drbd2165.out‚0' proc.memory.availableMB‚ 1409.609375! proc.memory.rssMB‚ 2896.578125 disk.rbd1.in‚0 disk.drbd2165.in‚0ÅÌËô2ƒ- _wandbruntime‚6334  _runtime‚6334s.i×=„Ñ{ =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7950869798660278 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8518518209457397 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3124851584434509 particle/accuracy_1_of_3‚1 'particle/f_score‚0.6396938562393188 6validation_average_jet_accuracy‚0.3468020558357239 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7389315366744995 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974368810653687 7"Purity/1FRt0SRqqt0FBt/event_purity‚0.30899578332901 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916567325592 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1964666247367859 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1917010098695755 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.48993250727653503 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8639978766441345 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.634249746799469 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.31914445757865906 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 particle/accuracy_1_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.050635166466236115 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_5_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9940261840820312 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_1_of_1‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6851851940155029 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3194161057472229 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951962232589722 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4643762707710266 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3146112859249115 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2937145531177521 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5303926467895508 +particle/sensitivity‚0.6366243362426758 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5167989730834961 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9884645342826843 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7616292238235474 *jet/accuracy_1_of_4‚0.9670838117599487 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.30808714032173157 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_4_of_5‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 *jet/accuracy_3_of_4‚0.0758780762553215 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2740442156791687 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1948685348033905 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1917010098695755 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9965658783912659 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.543749988079071 _step‚296 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4521337151527405 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9862009882926941 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.31949764490127563 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6042681932449341 particle/accuracy_1_of_2‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.49790796637535095 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5400711894035339 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_2_of_2‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6582331657409668 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5849682688713074 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5513381361961365 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.3505513370037079 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.19496166706085205 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2704945206642151 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.45845282077789307 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6851851940155029 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3064500391483307 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09688302129507065 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.694172203540802 *jet/accuracy_2_of_3‚0.5024328231811523 /particle/accuracy_3_of_4‚0.9968253374099731 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 jet/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5177844762802124 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9918069243431091 particle/accuracy_1_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.21296295523643494 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5774268507957458 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.3393876552581787 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 trainer/global_step‚3599 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4795970022678375 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.45845282077789307 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.4157239496707916 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9798122644424438 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚8.722782603235975 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9973413348197937 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974368810653687 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9724019169807434 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9032658338546753 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11662567406892776 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6372973322868347 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36011043190956116 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981483578681946 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7910053730010986 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5428685545921326 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.712196946144104 *jet/accuracy_2_of_4‚0.7669191956520081 'particle/accuracy‚0.846001386642456 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.28223946690559387 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942795634269714 7"Purity/0FRt2SRqqt*FBt/event_purity‚0.26286581158638 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23133423924446106 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.12980535626411438 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.3423803448677063 6 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.195875346660614 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6067019701004028 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9795351624488831 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6115140318870544 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9854400157928467 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5656218528747559 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5396138429641724 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8559553027153015 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.11881622672080994 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4386133551597595 /particle/accuracy_3_of_3‚0.9994933605194092 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9928118586540222 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3171622157096863 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6880725026130676 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2780606746673584 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5113824009895325 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.467110276222229 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.47345229983329773 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5881363153457642 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4628981649875641 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.511843740940094 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9962366223335266 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.4163685441017151 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.979678750038147 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9149966239929199 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.33051806688308716 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.07313350588083267 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9944894909858704 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04599056765437126 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5181747674942017 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6610302925109863 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7198790311813354 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.057534296065568924 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2998717427253723 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9861261248588562 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3171508312225342 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 +jet/accuracy_1_of_1‚0.30373668670654297 $ _timestamp‚1.784336173414895e+09 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5895069241523743 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1100553646683693 particle/accuracy_2_of_4‚1 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.992764413356781 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 5 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.30899578332901 *jet/accuracy_1_of_3‚0.9153517484664917 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458134651184 jet/accuracy_3_of_5‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5557564496994019 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 _runtime‚ 6334.0007674 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6636877655982971 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03426186367869377 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27652016282081604 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.30962875485420227 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9157307147979736 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.805649995803833 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9469974040985107 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5323615074157715 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6112309098243713 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 jet/accuracy_1_of_5‚1 epoch‚8 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8865960240364075 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7962396740913391 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6143779754638672 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.31943997740745544 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.519344687461853 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2393709123134613 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7843348979949951 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.694172203540802 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.21296295523643494 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6005940437316895 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9872214794158936 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.3099021315574646 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875â|hØ…Ö{ *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.48993250727653503 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8639978766441345 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.634249746799469 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.31914445757865906 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 particle/accuracy_1_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.050635166466236115 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_5_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9940261840820312 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_1_of_1‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6851851940155029 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3194161057472229 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951962232589722 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4643762707710266 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3146112859249115 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2937145531177521 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5303926467895508 +particle/sensitivity‚0.6366243362426758 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5167989730834961 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9884645342826843 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7616292238235474 *jet/accuracy_1_of_4‚0.9670838117599487 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.30808714032173157 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_4_of_5‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 *jet/accuracy_3_of_4‚0.0758780762553215 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2740442156791687 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1948685348033905 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1917010098695755 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9965658783912659 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.543749988079071 _step‚296 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4521337151527405 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9862009882926941 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.31949764490127563 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6042681932449341 particle/accuracy_1_of_2‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.49790796637535095 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5400711894035339 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_2_of_2‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6582331657409668 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5849682688713074 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5513381361961365 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.3505513370037079 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.19496166706085205 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2704945206642151 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.45845282077789307 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6851851940155029 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3064500391483307 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09688302129507065 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.694172203540802 *jet/accuracy_2_of_3‚0.5024328231811523 /particle/accuracy_3_of_4‚0.9968253374099731 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 jet/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5177844762802124 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9918069243431091 particle/accuracy_1_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.21296295523643494 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5774268507957458 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.3393876552581787 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 trainer/global_step‚3599 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4795970022678375 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.45845282077789307 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.4157239496707916 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9798122644424438 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚8.722782603235975 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9973413348197937 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974368810653687 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9724019169807434 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9032658338546753 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11662567406892776 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6372973322868347 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36011043190956116 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981483578681946 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7910053730010986 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5428685545921326 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7121969­ýºÎ!46144104 *jet/accuracy_2_of_4‚0.7669191956520081 'particle/accuracy‚0.846001386642456 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.28223946690559387 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942795634269714 7"Purity/0FRt2SRqqt*FBt/event_purity‚0.26286581158638 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23133423924446106 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.12980535626411438 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.3423803448677063 6 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.195875346660614 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6067019701004028 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9795351624488831 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6115140318870544 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9854400157928467 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5656218528747559 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5396138429641724 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8559553027153015 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.11881622672080994 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4386133551597595 /particle/accuracy_3_of_3‚0.9994933605194092 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9928118586540222 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3171622157096863 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6880725026130676 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2780606746673584 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5113824009895325 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.467110276222229 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.47345229983329773 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5881363153457642 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4628981649875641 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.511843740940094 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9962366223335266 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.4163685441017151 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.979678750038147 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9149966239929199 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.33051806688308716 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.07313350588083267 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9944894909858704 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04599056765437126 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5181747674942017 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6610302925109863 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7198790311813354 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.057534296065568924 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2998717427253723 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9861261248588562 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3171508312225342 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 +jet/accuracy_1_of_1‚0.30373668670654297 $ _timestamp‚1.784336173414895e+09 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5895069241523743 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1100553646683693 particle/accuracy_2_of_4‚1 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.992764413356781 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 5 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.30899578332901 *jet/accuracy_1_of_3‚0.9153517484664917 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458134651184 jet/accuracy_3_of_5‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5557564496994019 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 _runtime‚ 6334.0007674 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6636877655982971 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03426186367869377 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27652016282081604 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.30962875485420227 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9157307147979736 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.805649995803833 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9469974040985107 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5323615074157715 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6112309098243713 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 jet/accuracy_1_of_5‚1 epoch‚8 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8865960240364075 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7962396740913391 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6143779754638672 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.31943997740745544 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.519344687461853 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2393709123134613 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7843348979949951 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.694172203540802 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.21296295523643494 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6005940437316895 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9872214794158936 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.3099021315574646 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7950869798660278 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8518518209457397 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3124851584434509 particle/accuracy_1_of_3‚1 'particle/f_score‚0.6396938562393188 6validation_average_jet_accuracy‚0.3468020558357239 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7389315366744995 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974368810653687 7"Purity/1FRt0SRqqt0FBt/event_purity‚0.30899578332901 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916567325592 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1964666247367859 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1917010098695755¨¸ Ÿ†‚(:” ¸žëÒËØ·È gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚110.415' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.083000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OL™!ò‡‚(:ç ¸žëÒ’£ðÉ disk.drbd2165.in‚0* disk./.usagePercent‚39.357547403174266# proc.memory.rssMB‚ 2901.44921875 proc.cpu.threads‚17 network.sent‚ 6.874787e+06& disk.nvme0n1.out‚39624.62841796875 disk.sda.in‚ 0.47265625* proc.memory.percent‚17.709040641784668! disk.nvme0n1.in‚ 8037.19140625 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 8037.19140625( disk.nvme0c0n1.out‚39624.62841796875) proc.memory.availableMB‚ 2334.33203125 disk.drbd2165.out‚0 disk.dm-0.in‚ 8037.19140625 cpu‚25.532230669907367 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 255.27734375 disk.rbd1.out‚ 17.0546875$ memory_percent‚85.75236797332764# disk.dm-0.out‚39624.62841796875 network.recv‚ 1.594491e+06 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 255.27734375 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0 disk.sda.out‚ 255.27734375$ disk./.usageGB‚586.4558982849121 disk.rbd0.in‚0 disk.sda2.in‚0ÛÒ$锈‚(:‰ ÇžëÒºŒ§Ç gpu.0.gpu‚76.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚115.553' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%=Œæ‰‚(:Û ÇžëÒ……“Ê' disk.nvme0c0n1.out‚39628.2431640625% disk.nvme0n1.out‚39628.2431640625* proc.memory.percent‚17.709040641784668 disk.sda3.out‚ 255.82421875 disk.dm-0.in‚ 8097.578125# proc.memory.rssMB‚ 2901.44921875 network.recv‚ 1.599406e+06 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 255.82421875 cpu‚25.53187872234104 proc.cpu.threads‚17 network.sent‚ 6.93255e+06 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.47265625 disk.rbd1.in‚0 disk.rbd1.out‚ 17.0546875" disk.dm-0.out‚39628.2431640625$ memory_percent‚85.74976921081543( proc.memory.availableMB‚ 2334.7578125 disk.dm-1.in‚ 0.47265625* disk./.usagePercent‚39.357533578767416$ disk./.usageGB‚586.4556922912598 disk.sda2.out‚0 disk.sda.in‚ 0.47265625 disk.sda.out‚ 255.82421875 disk.nvme0n1.in‚ 8097.578125 disk.sda2.in‚0 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 8097.578125 disk.sda1.out‚0 ™ÿx”Š‚(:‰ ÖžëÒÀÛÃÈ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚288.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.7936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙLÂ/ï‹‚(:ä ÖžëÒîèÉ) disk./.usagePercent‚39.35730137993386# disk.nvme0c0n1.in‚ 8098.41015625 disk.dm-0.in‚ 8098.41015625# proc.memory.rssMB‚ 2901.44921875 disk.sda.in‚ 0.47265625 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 17.1328125 cpu‚25.531455136983126 network.sent‚ 6.937538e+06$ disk./.usageGB‚586.4522323608398 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0$ memory_percent‚85.75277328491211( disk.nvme0c0n1.out‚40608.18798828125 disk.sda2.out‚0 disk.sda.out‚ 256.50390625 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 256.50390625 disk.rbd1.in‚0* proc.memory.percent‚17.709040641784668 proc.cpu.threads‚17 disk.sda3.out‚ 256.50390625 disk.rbd0.in‚0 network.recv‚ 1.602796e+06# disk.dm-0.out‚40608.18798828125! disk.nvme0n1.in‚ 8098.41015625& disk.nvme0n1.out‚40608.18798828125' proc.memory.availableMB‚ 2334.265625<”ô”Œ‚(:‰ åžëÒ»Ç°Ç gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚279.906' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.9812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹ ªÑ‚(:Æ åžëҨܯÊ% disk.nvme0c0n1.out‚40615.58203125 cpu‚25.53114323116435 disk.sda1.out‚0 disk.nvme0n1.in‚8099.375 disk.dm-1.in‚ 0.47265625 disk.sda.in‚ 0.47265625 disk.sda.out‚ 256.796875 disk.dm-0.out‚40615.58203125$ memory_percent‚85.75241565704346) proc.memory.availableMB‚ 2334.32421875# proc.memory.rssMB‚ 2901.44921875* proc.memory.percent‚17.709040641784668 proc.cpu.threads‚17$ disk./.usageGB‚586.4520454406738 disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚8099.375) disk./.usagePercent‚39.35728883556468 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 256.796875 disk.sda2.in‚0 network.recv‚ 1.60635e+06# disk.nvme0n1.out‚40615.58203125 disk.sda2.out‚0 disk.dm-1.out‚ 256.796875 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚8099.375 network.sent‚ 6.942541e+06 disk.rbd1.in‚0 disk.rbd1.out‚ 17.1328125 disk.drbd2165.out‚0 disk.rbd0.in‚0#î3ŸŽ‚(:” ôžëÒʪÖÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.213' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.842600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d2†Ü‚(:Ñ ôžëÒ³ÇˆÊ disk.dm-1.in‚ 0.47265625 disk.drbd2165.in‚0 disk.sda3.in‚ 0.47265625 cpu‚25.530723101606714 disk.rbd0.in‚0 network.recv‚ 1.609561e+06 disk.sda1.out‚0" disk.dm-0.out‚40622.2099609375# memory_percent‚85.7539176940918 disk.rbd0.out‚0 disk.sda.out‚ 257.30078125# proc.memory.rssMB‚ 2901.44921875* proc.memory.percent‚17.709040641784668 disk.dm-0.in‚8099.375 disk.sda2.out‚0 network.sent‚ 6.947722e+06 disk.nvme0n1.in‚8099.375 disk.sda1.in‚0 disk.dm-1.out‚ 257.30078125 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625 disk.sda3.out‚ 257.30078125) disk./.usagePercent‚39.35729907586605$ disk./.usageGB‚586.4521980285645' disk.nvme0c0n1.out‚40622.2099609375 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd1.out‚ 17.1953125 disk.nvme0c0n1.in‚8099.375 disk.drbd2165.out‚0' proc.memory.availableMB‚ 2334.078125% disk.nvme0n1.out‚40622.2099609375PqÞ^”‚(:‰ ƒŸëÒäíÔÈ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚246.157' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.2314 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@&ùlë‘‚(:à ƒŸëÒ¯°ýÈ disk.dm-0.in‚ 8143.4140625 disk.rbd0.in‚0 disk.sda2.in‚0 network.sent‚ 6.952668e+06# disk.dm-0.out‚40630.26123046875 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 258.2578125 disk.dm-1.out‚ 258.2578125 proc.cpu.threads‚17 disk.sda.in‚ 0.47265625& disk.nvme0n1.out‚40630.26123046875 disk.sda1.out‚0 network.recv‚ 1.612849e+06* disk./.usagePercent‚39.357258370668106" disk.nvme0c0n1.in‚ 8143.4140625 disk.sda1.in‚0 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0# proc.memory.rssMB‚ 2901.44921875$ disk./.usageGB‚586.4515914916992 disk.nvme0n1.in‚ 8143.4140625* proc.memory.percent‚17.709040641784668( proc.memory.availableMB‚ 2334.5703125 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda.out‚ 258.2578125$ memory_percent‚85.75091361999512 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚40630.26123046875 disk.rbd1.out‚ 17.1953125 cpu‚25.530350340123324Q;h2’- _wandbruntime‚6422  _runtime‚6422–-&É“Ã _step‚297 ' lr-AdamW/pg1‚0.00045049452487013944 ' lr-AdamW/pg2‚0.00045049452487013944 trainer/global_step‚3649 % _timestamp‚1.7843362614749522e+09 _runtime‚6422.060058456;©¢£Î”È _runtime‚6422.060058456 _step‚297 ' lr-AdamW/pg1‚0.00045049452487013944 ' lr-AdamW/pg2‚0.00045049452487013944 trainer/global_step‚3649 % _timestamp‚1.7843362614749522e+09©ª\Qr2•- _wandbruntime‚6422  _runtime‚6422ì:·sý'–÷O ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1049241973891072 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31144319 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1052674915469824 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 _step‚298 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19754135 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3432941578752 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31607049 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚283450149613568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚283450517704704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4428740518400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9366416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚287879292759552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4428775054848 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31144319 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1052674915469824 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 _step‚298 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19754135 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3432941578752 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31607049 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚283450149613568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚283450517704704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4428740518400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9366416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚287879292759552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4428775054848 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62778045 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9368954 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1052955212805632 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1052953841235456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3429334286024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚3649 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚287957976184320 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1049519974621184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3433866614272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3435212011008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10390463 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62768052 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1049520000794624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1049066066345984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3427989685590 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31152525 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3433866614272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31625514 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31152531 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31615527 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4429972071936 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1052495400632008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10390222 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19759422 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31152525 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚283527636021248 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1052955212805632 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31615527 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19759176 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31152531 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _step‚299 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008/œ»q 'š†N C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62768052 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1052495400632008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10390222 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19759422 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31152525 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚283527636021248 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1052955212805632 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31615527 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19759176 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31152531 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _step‚299 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31625514 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1049519974621184 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62778045 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9368954 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1052955212805632 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1052953841235456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3429334286024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚3649 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚287957976184320 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1049519974621184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3433866614272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3435212011008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10390463 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62768052 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1049520000794624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1049066066345984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3427989685590 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31152525 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3433866614272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31625514 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31152531 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31615527 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4429972071936 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1066622639469056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31554619 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63597399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚6505.519040533 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10525269 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32032801 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3479257509376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1063142010396672 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1066621267906048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20018835 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3699 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3474647932774 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63587420 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3479257509376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1062682248048640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0õA“(­üO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9493566 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3473303332452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1063142010396672 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1066621267906048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20018835 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3699 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3474647932774 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63587420 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3479257509376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1062682248048640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31554625 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10525001 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4488562844160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 % _timestamp‚1.7843363449336863e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1066622639469056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31554619 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63597399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚6505.519040533 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10525269 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32032801 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3479257509376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63614097 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1066901565241856 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1066900193671680 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20024079 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3474226798698 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚303 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3480182544896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63614097 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3481527941632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32051266 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4489784019456 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287402324496384 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1066435726653916 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32041279 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3480182544896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9496104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚291891703798272 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32041279 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31562831 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20023810 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63604104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1066434356625514 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1062960155254784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1063420037300224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10527970 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4489747393024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚315628314Œ@¼à°†N T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1066434356625514 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1062960155254784 KEDeviceStatsMonitor.on_train_batиõw,&ch_end/reserved_bytes.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1063420037300224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10527970 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4489747393024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31562831 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31562825 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63614097 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1066901565241856 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1066900193671680 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20024079 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3474226798698 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚303 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3480182544896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63614097 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3481527941632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32051266 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4489784019456 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287402324496384 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1066435726653916 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32041279 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3480182544896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9496104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚291891703798272 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32041279 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31562831 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20023810 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63604104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820¯TS2±- _wandbruntime‚6507  _runtime‚6507vcs˜²‚ 2loss/SRqqt2/assignment_loss‚0.5364986890961091 /loss/FRt2/detection_loss‚1.6383827825817046 trainer/global_step‚3699 _runtime‚6507.314430805 _step‚304 /loss/FBt2/detection_loss‚0.9478026942440099 %loss/total_loss‚8.732934141174441 epoch‚9 /loss/FRt2/assignment_loss‚0.822272401578671 2loss/SRqqt1/assignment_loss‚0.5424280246306302 2loss/FBt1/assignment_loss‚0.005987857813552109 /loss/FRt1/detection_loss‚0.5145962166120337 % _timestamp‚1.7843363467292407e+09 1loss/SRqqt2/detection_loss‚0.6941341895431965 0loss/FRt1/assignment_loss‚1.7585165135668352 1loss/FBt2/assignment_loss‚0.03378483158249149 1loss/SRqqt1/detection_loss‚0.7637896847411478 0loss/FBt1/detection_loss‚0.47474025518406093îÜ ³‡ % _timestamp‚1.7843363467292407e+09 1loss/SRqqt2/detection_loss‚0.6941341895431965 0loss/FRt1/assignment_loss‚1.7585165135668352 1loss/FBt2/assignment_loss‚0.03378483158249149 1loss/SRqqt1/detection_loss‚0.7637896847411478 0loss/FBt1/detection_loss‚0.47474025518406093 2loss/SRqqt2/assignment_loss‚0.5364986890961091 /loss/FRt2/detection_loss‚1.6383827825817046 trainer/global_step‚3699 _runtime‚6507.314430805 _step‚304 /loss/FBt2/detection_loss‚0.9478026942440099 %loss/total_loss‚8.732934141174441 epoch‚9 /loss/FRt2/assignment_loss‚0.822272401578671 2loss/SRqqt1/assignment_loss‚0.5424280246306302 2loss/FBt1/assignment_loss‚0.005987857813552109 /loss/FRt1/detection_loss‚0.5145962166120337°+É s”´‚(:‰ ÝŸëÒàªìÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚265.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð·“쵂(:á ÝŸëÒÄÚÂÉ% disk.nvme0n1.out‚40699.2724609375 disk.sda3.out‚ 267.68359375* proc.memory.percent‚17.709040641784668 proc.cpu.threads‚17 network.recv‚ 1.634411e+06 disk.sda3.in‚ 0.47265625 disk.rbd0.out‚0$ memory_percent‚91.68658256530762) disk./.usagePercent‚39.35736666185509 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 8224.01953125 disk.rbd0.in‚0 network.sent‚ 7.039783e+06 disk.sda2.out‚0 disk.dm-1.out‚ 267.68359375# disk.nvme0c0n1.in‚ 8224.01953125 disk.sda.out‚ 267.68359375 disk.sda1.in‚0 cpu‚25.52828300881848 disk.rbd1.out‚ 17.4140625 disk.sda1.out‚0( proc.memory.availableMB‚ 1362.0703125$ disk./.usageGB‚586.4532051086426 disk.dm-1.in‚ 0.47265625 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0" disk.dm-0.out‚40699.2724609375# proc.memory.rssMB‚ 2901.44921875' disk.nvme0c0n1.out‚40699.2724609375 disk.dm-0.in‚ 8224.019531254½clŸ¶‚(:” ìŸëÒŽ¨šÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚274.213' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.842600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Åi)5ì·‚(:á ìŸëÒåÕƒÉ disk.sda2.out‚0# disk.nvme0c0n1.in‚ 8224.01953125 network.sent‚ 7.102898e+06$ disk./.usageGB‚586.4531517028809 disk.drbd2165.in‚0 disk.dm-0.in‚ 8224.01953125 network.recv‚ 1.638978e+06 disk.rbd0.out‚0 disk.drbd2165.out‚0) proc.memory.percent‚17.77641773223877' disk.nvme0c0n1.out‚40708.4990234375% disk.nvme0n1.out‚40708.4990234375 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda.in‚ 0.47265625 disk.sda.out‚ 268.77734375) disk./.usagePercent‚39.35736307774961! disk.nvme0n1.in‚ 8224.01953125 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 cpu‚25.527916894389104 disk.sda3.out‚ 268.77734375 disk.dm-1.in‚ 0.47265625 disk.rbd1.out‚ 17.5390625# proc.memory.rssMB‚ 2912.48828125 disk.dm-1.out‚ 268.77734375" disk.dm-0.out‚40708.4990234375 proc.cpu.threads‚17# memory_percent‚91.6672945022583) proc.memory.availableMB‚ 1365.23046875 disk.sda3.in‚ 0.47265625T²HΔ¸‚(:‰ ûŸëÒåÃÝÈ gpu.0.gpu‚87.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚268.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ™ç¼ã¹‚(:Ø ûŸëÒ¢À¿É proc.memory.rssMB‚ 2917.3125) disk./.usagePercent‚39.35742835967084 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 8224.984375 disk.sda1.in‚0* proc.memory.percent‚17.805862426757812$ memory_percent‚91.67084693908691 disk.rbd1.out‚ 17.5390625 disk.sda.in‚ 0.47265625 disk.sda.out‚ 269.3515625( disk.nvme0c0n1.out‚40715.87548828125 disk.nvme0n1.in‚ 8224.984375 cpu‚25.527544684579407& disk.nvme0n1.out‚40715.87548828125( proc.memory.availableMB‚ 1364.6484375 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 8224.984375 disk.dm-1.out‚ 269.3515625# disk.dm-0.out‚40715.87548828125 network.recv‚ 1.642493e+06 disk.sda3.in‚ 0.47265625 disk.dm-1.in‚ 0.47265625 disk.drbd2165.out‚0 network.sent‚ 7.107764e+06$ disk./.usageGB‚586.4541244506836 disk.sda3.out‚ 269.3515625 disk.rbd0.out‚0 disk.sda2.out‚0Ûö¸!Ÿº‚(:” Š ëÒæ…ÙÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚267.197' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.439400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xÖçÉໂ(:Õ Š ëÒ™¨àÉ network.recv‚ 1.645717e+06 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 1364.3984375 disk.sda3.out‚ 269.73828125 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625 disk.dm-0.in‚ 8288.2890625 disk.dm-0.out‚40727.78515625 disk.rbd1.out‚ 17.6171875 disk.drbd2165.in‚0 proc.memory.rssMB‚ 2917.3125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.out‚ 269.73828125 network.sent‚ 7.112793e+06$ disk./.usageGB‚586.4526062011719 disk.drbd2165.out‚0* proc.memory.percent‚17.805862426757812" disk.nvme0c0n1.in‚ 8288.2890625$ memory_percent‚91.67237281799316 cpu‚25.527175921744096) disk./.usagePercent‚39.35732646867221 disk.dm-1.out‚ 269.73828125# disk.nvme0n1.out‚40727.78515625 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 disk.dm-1.in‚ 0.47265625% disk.nvme0c0n1.out‚40727.78515625 disk.nvme0n1.in‚ 8288.2890625°ËŸ¼‚(:” ™ ëÒ‘ëÇÈ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚284.671' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.934200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ7î+ݽ‚(:Ò ™ ëÒËüÃÉ disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 270.09375" disk.nvme0c0n1.in‚ 8288.2890625* proc.memory.percent‚17.805862426757812 network.recv‚ 1.649442e+06 disk.sda2.in‚0 disk.dm-0.in‚ 8288.2890625 disk.nvme0n1.in‚ 8288.2890625 network.sent‚ 7.117897e+06$ memory_percent‚91.67237281799316 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 17.6171875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 270.09375 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda.in‚ 0.47265625( proc.memory.availableMB‚ 1364.3984375 proc.memory.rssMB‚ 2917.3125$ disk./.usageGB‚586.4524383544922% disk.nvme0n1.out‚40731.6025390625 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 cpu‚25.526812585138103' disk.nvme0c0n1.out‚40731.6025390625" disk.dm-0.out‚40731.6025390625 disk.sda.out‚ 270.09375) disk./.usagePercent‚39.35731520434071O S”¾‚(:‰ ¨ ëÒ‡›´È gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚281.671' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ýd—ïã¿‚(:Ø ¨ ëÒ¤Š§É$ disk./.usageGB‚586.4946212768555 disk.sda3.out‚ 272.00390625 disk.rbd0.out‚0 disk.sda.out‚ 272.00390625 proc.cpu.threads‚17 network.recv‚ 1.652739e+06$ disk.nvme0n1.out‚40770.166015625 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625& disk.nvme0c0n1.out‚40770.166015625* proc.memory.percent‚17.805862426757812 network.sent‚ 7.122872e+06 disk.rbd1.out‚ 17.6796875" disk.nvme0c0n1.in‚ 8289.2578125) disk./.usagePercent‚39.36014613565434 disk.sda2.out‚0 disk.dm-1.out‚ 272.00390625 disk.sda.in‚ 0.47265625) proc.memory.availableMB‚ 1365.26953125 proc.memory.rssMB‚ 2917.3125 cpu‚25.526451051220704 disk.dm-0.in‚ 8289.2578125! disk.dm-0.out‚40770.166015625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 8289.2578125 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0# memory_percent‚91.6670560836792 disk.sda1.out‚0úκ²2À- _wandbruntime‚6589  _runtime‚6589|–ÇÁÁ % _timestamp‚1.7843364284268122e+09 _runtime‚6589.011922294 _step‚305 & lr-AdamW/pg1‚0.0004485438819907781 & lr-AdamW/pg2‚0.0004485438819907781 trainer/global_step‚3749Ûi¶‰ÌÂÆ _runtime‚6589.011922294 _step‚305 & lr-AdamW/pg1‚0.0004485438819907781 & lr-AdamW/pg2‚0.0004485438819907781 trainer/global_step‚3749 % _timestamp‚1.7843364284268122e+09±Ž¹#ø2Ã- _wandbruntime‚6589  _runtime‚6589Ü`0žý'Ä÷O _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4548040680960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1077042046902272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4548077314560 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64433451 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32468526 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1077042046902272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚291198790381568 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1076576336957440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3520885045882 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1080095877403000 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1077042073075712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31964925 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9620716 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3525573440000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10661854 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1080567620342272 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 % _timestamp‚1.7843364284280858e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32458553 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _step‚306 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3519540445560 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3526918829568 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9620711 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291199158472704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64423472 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32468526 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1080097247431290 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1080568991905280 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1080567620342272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚6589.013433507 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3525573440000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚3749 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20282840 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1080568991905280 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31964925 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31964919 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚295746831062528 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32458553 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295747235787264 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1077042073075712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31964919 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1080567620342272 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 % _timestamp‚1.7843364284280858e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32458553 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _step‚306 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3519540445560 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3526918829568 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9620711 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291199158472704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64423472 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32468526 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1080097247431290 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1080568991905280 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1080567620342272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚6589.013433507 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3525573440000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚3749 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20282840 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1080568991905280 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31964925 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31964919 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚295746831062528 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32458553 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295747235787264 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1077042073075712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640²Ç) 2Æ- _wandbruntime‚6590  _runtime‚6590C‡xÍ'ÇN NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31973131 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31973131 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1080846546107904 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1077320073805824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295825505425920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20288034 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3521808512240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31973125 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20287769 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1077320073805824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64450149 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚6590.439758457 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32477018 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1076854218735616 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64440156 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚307 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9623259 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3526498475520 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1076854244163584 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1080847917678080 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3526498475520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3527843872256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1077320047632384 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9623254 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚291276644880384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 % _timestamp‚1.7843364298541884e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64440156 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ï˜}t 'ȆN \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3527843872256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1077320047632384 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9623254 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚291276644880384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 % _timestamp‚1.7843364298541884e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64440156 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31973131 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31973131 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1080846546107904 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1077320073805824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295825505425920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20288034 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3521808512240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31973125 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20287769 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1077320073805824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64450149 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚6590.439758457 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32477018 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1076854218735616 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64440156 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚307 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9623259 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3526498475520 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1076854244163584 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1080847917678080 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3526498475520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648³xk¤›2É- _wandbruntime‚6590  _runtime‚6590„¡Ê€ 2loss/SRqqt1/assignment_loss‚0.5196058733964575 %loss/total_loss‚9.253029937871194 1loss/FBt2/assignment_loss‚0.04181090435906398 1loss/SRqqt2/detection_loss‚0.7082460794411892 /loss/FBt1/detection_loss‚0.7735193123012055 _runtime‚6590.505231817 0loss/FRt1/assignment_loss‚1.7456174483330567 /loss/FRt2/assignment_loss‚0.897794196382165 2loss/SRqqt2/assignment_loss‚0.5431784492195804 /loss/FRt1/detection_loss‚0.5102217342230655 /loss/FBt2/detection_loss‚0.9979004140186589 trainer/global_step‚3749 $ _timestamp‚1.784336429920214e+09 _step‚308 2loss/FBt1/assignment_loss‚0.010478769349372589 /loss/FRt2/detection_loss‚1.7884779583400814 1loss/SRqqt1/detection_loss‚0.7161787985072982 epoch‚9Ê°¯ Ë… 1loss/FBt2/assignment_loss‚0.04181090435906398 1loss/SRqqt2/detection_loss‚0.7082460794411892 /loss/FBt1/detection_loss‚0.7735193123012055 _runtime‚6590.505231817 0loss/FRt1/assignment_loss‚1.7456174483330567 /loss/FRt2/assignment_loss‚0.897794196382165 2loss/SRqqt2/assignment_loss‚0.5431784492195804 /loss/FRt1/detection_loss‚0.5102217342230655 /loss/FBt2/detection_loss‚0.9979004140186589 trainer/global_step‚3749 $ _timestamp‚1.784336429920214e+09 _step‚308 2loss/FBt1/assignment_loss‚0.010478769349372589 /loss/FRt2/detection_loss‚1.7884779583400814 1loss/SRqqt1/detection_loss‚0.7161787985072982 epoch‚9 2loss/SRqqt1/assignment_loss‚0.5196058733964575 %loss/total_loss‚9.253029937871194´Ç›´”Ì‚(:‰ · ëÒÂÕ¹Ç gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚281.671' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉIgIàÍ‚(:Õ · ëÒ›„‰É network.recv‚ 1.656346e+06 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.out‚40778.26953125 proc.memory.rssMB‚ 2917.3125 disk.nvme0n1.in‚ 8289.2734375# disk.nvme0n1.out‚40778.26953125 disk.sda3.in‚ 0.47265625 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625% disk.nvme0c0n1.out‚40778.26953125 cpu‚25.5260911358068 network.sent‚ 7.127983e+06" disk.nvme0c0n1.in‚ 8289.2734375 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1365.26171875* proc.memory.percent‚17.805862426757812$ disk./.usageGB‚586.6665077209473 disk.sda3.out‚ 272.61328125 disk.rbd1.in‚0 disk.rbd1.out‚ 17.6796875$ memory_percent‚91.66710376739502 disk.sda.out‚ 272.61328125 proc.cpu.threads‚17 disk.dm-1.out‚ 272.61328125 disk.drbd2165.out‚0* disk./.usagePercent‚39.371681579139675 disk.sda1.in‚0 disk.dm-0.in‚ 8289.2734375,3?—žÎ‚(:“ Æ ëÒòôÑÇ gpu.0.gpu‚83.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚258.285' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.65700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f„£äÏ‚(:Ù Æ ëÒÿ§êÊ* proc.memory.percent‚17.805862426757812) proc.memory.availableMB‚ 1364.98046875 disk.sda2.in‚0 network.sent‚ 7.191237e+06 disk.nvme0n1.in‚ 8338.734375 disk.rbd0.out‚0 disk.rbd1.out‚ 17.7578125 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625! disk.nvme0c0n1.in‚ 8338.734375( disk.nvme0c0n1.out‚40907.60400390625 disk.dm-0.in‚ 8338.734375 proc.cpu.threads‚17 disk.sda1.in‚0# disk.dm-0.out‚40907.60400390625& disk.nvme0n1.out‚40907.60400390625 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0 disk.drbd2165.out‚0$ memory_percent‚91.66882038116455 proc.memory.rssMB‚ 2917.3125 cpu‚25.525717009575526 disk.drbd2165.in‚0 disk.sda3.out‚ 273.7109375 disk.rbd0.in‚0 disk.sda.out‚ 273.7109375 disk.dm-1.out‚ 273.7109375 disk.sda.in‚ 0.47265625 network.recv‚ 1.661142e+06) disk./.usagePercent‚39.37788284964157$ disk./.usageGB‚586.7589111328125(Õº$7Ђ(:‰ Õ ëÒ…Õ¶Ç gpu.0.gpu‚87.0 gpu.0.˜53Ö]memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚288.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.6492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æÈ*íÑ‚(:â Õ ëÒ¹ÐÉ$ disk./.usageGB‚588.8752593994141 proc.cpu.threads‚17$ memory_percent‚91.66882038116455 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625 disk.sda.out‚ 274.14453125 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 8339.69921875& disk.nvme0n1.out‚41142.44580078125 disk.dm-0.in‚ 8339.69921875 disk.sda3.in‚ 0.47265625 disk.dm-1.in‚ 0.47265625* proc.memory.percent‚17.805862426757812) disk./.usagePercent‚39.51991275754768# disk.dm-0.out‚41142.44580078125 disk.sda2.out‚0 network.recv‚ 1.664634e+06 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 8339.69921875 disk.rbd0.in‚0 disk.dm-1.out‚ 274.14453125 network.sent‚ 7.196196e+06 disk.sda1.out‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1364.98046875 disk.rbd1.out‚ 17.7578125( disk.nvme0c0n1.out‚41142.44580078125 disk.sda1.in‚0 disk.sda3.out‚ 274.14453125 disk.rbd0.out‚0 proc.memory.rssMB‚ 2917.3125 cpu‚25.525368020795934~ú¥LžÒ‚(:“ ä ëÒìÙ×È gpu.0.gpu‚84.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚262.648' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.52960000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒB=wÛÓ‚(:Ð ä ëÒïðúÈ disk.nvme0n1.in‚ 8339.703125) proc.memory.availableMB‚ 1365.18359375 disk.dm-1.out‚ 274.65625 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625" disk.dm-0.out‚43242.4833984375% disk.nvme0n1.out‚43243.6162109375* proc.memory.percent‚17.805862426757812) disk./.usagePercent‚39.52675225483244$ disk./.usageGB‚588.9771728515625 disk.rbd1.out‚ 17.8203125 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚43243.6162109375 proc.memory.rssMB‚ 2917.3125 cpu‚25.525016505237947 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚91.66758060455322 network.sent‚ 7.201747e+06 network.recv‚ 1.667955e+06 disk.dm-1.in‚ 0.47265625 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0 disk.dm-0.in‚ 8339.703125 disk.sda.out‚ 274.65625 disk.sda1.out‚0 disk.sda3.out‚ 274.65625 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 8339.703125ýì'ÄŸÔ‚(:” ó ëÒ§‘Ç gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚271.506' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.301199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=.2îÕ‚(:ã ó ëÒÄ–õÉ network.recv‚ 1.671353e+06* disk./.usagePercent‚39.553627413761916 disk.dm-0.in‚ 8340.53515625 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625 proc.memory.rssMB‚ 2917.3125$ disk./.usageGB‚589.3776321411133# disk.nvme0c0n1.in‚ 8340.53515625# disk.dm-0.out‚43516.75439453125) proc.memory.availableMB‚ 1365.19921875 disk.sda.out‚ 277.53515625& disk.nvme0n1.out‚43516.75439453125 disk.sda3.out‚ 277.53515625 disk.rbd1.in‚0 disk.rbd1.out‚ 17.8203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚43516.75439453125 disk.sda2.in‚0 disk.dm-1.out‚ 277.53515625* proc.memory.percent‚17.805862426757812 cpu‚25.524653538210504 proc.cpu.threads‚17 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.47265625 disk.rbd0.out‚0$ memory_percent‚91.66748523712158 network.sent‚ 7.208278e+06! disk.nvme0n1.in‚ 8340.53515625g>$ê2Ö- _wandbruntime‚6672  _runtime‚6672÷ƒ·É×à ' lr-AdamW/pg1‚0.00044754855658519745 ' lr-AdamW/pg2‚0.00044754855658519745 trainer/global_step‚3799 % _timestamp‚1.7843365118345106e+09 _runtime‚6672.419596019 _step‚309™çyÎØÈ % _timestamp‚1.7843365118345106e+09 _runtime‚6672.419596019 _step‚309 ' lr-AdamW/pg1‚0.00044754855658519745 ' lr-AdamW/pg2‚0.00044754855658519745 trainer/global_step‚3799µ‚’©Q2Ù- _wandbruntime‚6672  _runtime‚6672£2´ý'Ú÷O F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3573234760192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32375225 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _runtime‚6672.421192603 _step‚310 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1094513972778496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65269503 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32375225 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _runtime‚6672.421192603 _step‚310 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1094513972778496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65269503 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65286201 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32383431 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32902770 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1091220110311424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚299759441936896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3574159802880 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1094315008669330 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1091220084137984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10802250 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚295150965264384 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3566701024914 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20552654 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9750409 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1090748333072384 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1090748307644416 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚65276208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4608844763648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32383431 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1094794270114304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32383425 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3568045625348 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3572814406144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3572814406144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1094794270114304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _step‚311 \GDeviceStats»–(±ä Monitor.on_train_batch_end/active_bytes.large_pool.allocated‚1091220110311424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3574159802880 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚295150597173248 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65286201 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1094316378697732 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1091220084137984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20552919 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65276208 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 $ _timestamp‚1.784336513259713e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1094794270114304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _step‚311 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1091220110311424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3574159802880 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚295150597173248 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65286201 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1094316378697732 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1091220084137984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20552919 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65276208 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 $ _timestamp‚1.784336513259713e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65286201 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32383431 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32902770 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1091220110311424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚299759441936896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3574159802880 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1094315008669330 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1091220084137984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10802250 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚295150965264384 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3566701024914 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20552654 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9750409 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1090748333072384 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1090748307644416 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚65276208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4608844763648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32383431·ºwûÖ2ß- _wandbruntime‚6673  _runtime‚6673«.àü epoch‚9 _step‚312 0loss/FRt1/assignment_loss‚1.7093532757749985 1loss/FBt1/assignment_loss‚0.02111156478503085 trainer/global_step‚3799 /loss/FRt2/detection_loss‚1.6410409388680653 1loss/SRqqt1/detection_loss‚0.7812049895543937 /loss/SRqqt2/detection_loss‚0.69956973380047 /loss/FBt2/detection_loss‚0.7695469288776319 0loss/FRt2/assignment_loss‚0.8856296442774697 2loss/SRqqt1/assignment_loss‚0.5423212711968572 -loss/FBt1/detection_loss‚0.42015183146394 %loss/total_loss‚8.629560038202616 % _timestamp‚1.7843365133223596e+09 _runtime‚6673.907566248 2loss/SRqqt2/assignment_loss‚0.5186749340033107 0loss/FBt2/assignment_loss‚0.0710895026963802 /loss/FRt1/detection_loss‚0.5698654229040676SÒÅšá 0loss/FRt2/assignment_loss‚0.8856296442774697 2loss/SRqqt1/assignment_loss‚0.5423212711968572 -loss/FBt1/detection_loss‚0.42015183146394 %loss/total_loss‚8.629560038202616 % _timestamp‚1.7843365133223596e+09 _runtime‚6673.907566248 2loss/SRqqt2/assignment_loss‚0.5186749340033107 0loss/FBt2/assignment_loss‚0.0710895026963802 /loss/FRt1/detection_loss‚0.5698654229040676 epoch‚9 _step‚312 0loss/FRt1/assignment_loss‚1.7093532757749985 1loss/FBt1/assignment_loss‚0.02111156478503085 trainer/global_step‚3799 /loss/FRt2/detection_loss‚1.6410409388680653 1loss/SRqqt1/detection_loss‚0.7812049895543937 /loss/SRqqt2/detection_loss‚0.69956973380047 /loss/FBt2/detection_loss‚0.7695469288776319¸t„JŸâ‚(:” ‚¡ëÒ»üÉÇ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚249.151' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.830200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iܸèã‚(:Ý ‚¡ëÒë«÷É( disk.nvme0c0n1.out‚43572.08154296875 disk.dm-1.out‚ 279.64453125 cpu‚25.524299722882535 network.recv‚ 1.674846e+06 disk.dm-0.in‚ 8392.57421875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda.out‚ 279.64453125' proc.memory.availableMB‚ 1364.671875 network.sent‚ 7.2112e+06& disk.nvme0n1.out‚43572.08154296875 disk.sda3.out‚ 279.64453125 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 8392.57421875 disk.sda.in‚ 0.47265625 proc.memory.rssMB‚ 2917.3125* proc.memory.percent‚17.805862426757812 disk.drbd2165.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 disk.rbd1.out‚ 17.9609375$ memory_percent‚91.67070388793945) disk./.usagePercent‚39.55651953087623# disk.dm-0.out‚43572.08154296875 disk.rbd0.in‚0# disk./.usageGB‚589.420726776123# disk.nvme0c0n1.in‚ 8392.57421875tEºûžä‚(:“ ‘¡ëÒ¼Ù™Ç gpu.0.gpu‚80.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m°A9ëå‚(:à ‘¡ëÒ¾«úÈ network.recv‚ 1.679817e+06$ disk./.usageGB‚589.3021087646484 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 280.28515625# disk.dm-0.out‚43861.80419921875$ memory_percent‚91.67261123657227 disk.sda.in‚ 0.47265625* proc.memory.percent‚17.805862426757812 disk.rbd0.out‚0( disk.nvme0c0n1.out‚43861.80419921875 cpu‚25.523918201626753) disk./.usagePercent‚39.54855897659903 disk.dm-1.out‚ 280.28515625 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 0.47265625! disk.nvme0n1.in‚ 8392.57421875' proc.memory.availableMB‚ 1364.359375 proc.cpu.threads‚17 disk.sda.out‚ 280.28515625# disk.nvme0c0n1.in‚ 8392.57421875 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 17.9609375 disk.dm-0.in‚ 8392.57421875 proc.memory.rssMB‚ 2917.3125 disk.sda2.out‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚43861.80419921875 disk.sda1.out‚0 network.sent‚ 7.274274e+06_C”æ‚(:‰  ¡ëÒ£âòÈ gpu.0.gpu‚87.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚291.723' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.3446 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JkX_äç‚(:Ù  ¡ëÒØËÉ* proc.memory.percent‚17.805862426757812 disk.rbd1.in‚0 cpu‚25.523562695895144 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.rbd1.out‚ 18.0234375 proc.memory.rssMB‚ 2917.3125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 8393.40625 disk.sda3.out‚ 280.77734375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 8393.40625$ memory_percent‚91.67139530181885) proc.memory.availableMB‚ 1364.55859375 network.sent‚ 7.279501e+06 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda.in‚ 0.47265625 network.recv‚ 1.68353e+06* disk./.usagePercent‚39.547670118440145 disk.rbd0.out‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚43879.84619140625& disk.nvme0n1.out‚43879.84619140625 disk.dm-1.out‚ 280.77734375 disk.sda.out‚ 280.77734375# disk.dm-0.out‚43879.84619140625 disk.nvme0n1.in‚ 8393.40625$ disk./.usageGB‚589.2888641357422 disk.dm-1.in‚ 0.47265625<|1”è‚(:‰ ¯¡ëÒ¶•“È gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚247.557' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.5114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©\Äáé‚(:Ö ¯¡ëÒðãçÈ$ disk./.usageGB‚589.2898712158203 disk.dm-1.out‚ 281.1328125 disk.sda.in‚ 0.47265625 disk.sda.out‚ 281.1328125 cpu‚25.52325699540699 proc.cpu.threads‚17 disk.dm-1.in‚ 0.47265625* proc.memory.percent‚17.805862426757812 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 8394.37109375 disk.sda1.in‚0 proc.memory.rssMB‚ 2917.3125# disk.nvme0c0n1.in‚ 8394.37109375 disk.sda3.out‚ 281.1328125 disk.sda3.in‚ 0.47265625 network.sent‚ 7.284374e+06) disk./.usagePercent‚39.54773770442919 disk.sda1.out‚0 disk.rbd1.in‚0" proc.memory.availableMB‚1364.5# disk.dm-0.out‚43890.37158203125# memory_percent‚91.6717529296875( disk.nvme0c0n1.out‚43890.37158203125& disk.nvme0n1.out‚43890.37158203125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 18.0234375 network.recv‚ 1.687192e+06 disk.dm-0.in‚ 8394.37109375 disk.sda2.in‚0 disk.drbd2165.out‚0°9Ïߟê‚(:” ¾¡ëÒɵ¥Ç gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚279.015' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.803000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡IG;Üë‚(:Ñ ¾¡ëÒ²ˆÏÉ disk.rbd1.in‚0 disk.sda.out‚ 281.34375 disk.drbd2165.in‚0 disk.dm-0.in‚ 8458.77734375* proc.memory.percent‚17.805862426757812 network.sent‚ 7.289485e+06 disk.sda3.in‚ 0.47265625 disk.sda2.in‚0 disk.dm-1.out‚ 281.34375( disk.nvme0c0n1.out‚43896.38525390625$ disk./.usageGB‚589.2902870178223! disk.nvme0n1.in‚ 8458.77734375& disk.nvme0n1.out‚43896.38525390625 disk.dm-1.in‚ 0.47265625# disk.nvme0c0n1.in‚ 8458.77734375 network.recv‚ 1.690386e+06 disk.rbd1.out‚ 18.1015625# memory_percent‚91.6717529296875 disk.sda1.in‚0 disk.sda3.out‚ 281.34375 cpu‚25.522866156328995) disk./.usagePercent‚39.54776560925042 disk.sda2.out‚0 disk.sda.in‚ 0.47265625# disk.dm-0.out‚43896.38525390625" proc.memory.availableMB‚1364.5 disk.sda1.out‚0 disk.rbd0.in‚0 disk.drbd2165.out‚0 proc.memory.rssMB‚ 2917.3125 proc.cpu.threads‚17 disk.rbd0.out‚0˜•ö”ì‚(:‰ Í¡ëÒîôÃÇ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚293.271' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.6542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H²´÷çí‚(:Ü Í¡ëÒÕêÊ network.sent‚ 7.296533e+06 disk.dm-0.in‚ 8459.609375! disk.nvme0c0n1.in‚ 8459.609375$ memory_percent‚91.67301654815674 network.recv‚ 1.693872e+06$ disk./.usageGB‚589.3314552307129# disk.dm-0.out‚43904.21728515625 disk.rbd1.out‚ 18.1015625 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1364.29296875 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 281.57421875 disk.sda3.out‚ 281.57421875 disk.dm-1.in‚ 0.47265625( disk.nvme0c0n1.out‚43904.21728515625 proc.memory.rssMB‚ 2917.3125* proc.memory.percent‚17.805862426757812 cpu‚25.522547765237757 proc.cpu.threads‚17& disk.nvme0n1.out‚43904.21728515625 disk.drbd2165.in‚0) disk./.usagePercent‚39.55052844255994 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.rbd0.out‚0 disk.dm-1.out‚ 281.57421875 disk.nvme0n1.in‚ 8459.609375 disk.sda1.out‚0‡oJ†2î- _wandbruntime‚6755  _runtime‚6755e:•OÉïà _runtime‚6755.644278793 _step‚313 ' lr-AdamW/pg1‚0.00044653998015447324 ' lr-AdamW/pg2‚0.00044653998015447324 trainer/global_step‚3849 % _timestamp‚1.7843365950591452e+09VJˆåÎðÈ _runtime‚6755.644278793 _step‚313 ' lr-AdamW/pg1‚0.00044653998015447324 ' lr-AdamW/pg2‚0.00044653998015447324 trainer/global_step‚3849 % _timestamp‚1.7843365950591452e+09¹x¥2ñ- _wandbruntime‚6755  _runtime‚6755ŽQò›N%ò÷O A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33310057 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20813101 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1104842119913472 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20813355 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33310057 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10938090 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33320030 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚298947431149568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3618205301248 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10938339 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32785519 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32785525 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _runtime‚6755.645971248 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66095576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9875011 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298947799240704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3612014671776 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33320030 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32785525 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1108460325214720 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1104842146086912 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32785519 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3619550690816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3619550690816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3618205301248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3613359272098 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚303615591959040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚303615189331456 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚3849 % _timestamp‚1.7843365950605693e+09 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1104842119913472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚F9ñů 4667758181888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1104364514775040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1107976529446816 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4667792718336Bõm(óüO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20813101 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1104842119913472 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20813355 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33310057 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10938090 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33320030 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚298947431149568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3618205301248 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10938339 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32785519 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32785525 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _runtime‚6755.645971248 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66095576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9875011 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298947799240704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3612014671776 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33320030 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32785525 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1108460325214720 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1104842146086912 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32785519 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3619550690816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3619550690816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3618205301248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3613359272098 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚303615591959040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚303615189331456 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚3849 % _timestamp‚1.7843365950605693e+09 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1104842119913472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4667758181888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1104364514775040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1107976529446816 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4667792718336 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33310057 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10941089 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10940849 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 trainer/global_step‚3849 _runtime‚6757.077982109 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66112260 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66122253 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚303694273541632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1105120120643584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3619130336768 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1105120146817024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33328522 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20818403 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 % _timestamp‚1.7843365964925575e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚303693870921216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4668987893248 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3620475733504 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32793731 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66112260 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3612938138022 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299024917557248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434.|• 'ö†N OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3612938138022 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299024917557248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1104642396553216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32793731 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10941089 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10940849 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 trainer/global_step‚3849 _runtime‚6757.077982109 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66112260 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66122253 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚303694273541632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1105120120643584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3619130336768 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1105120146817024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33328522 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20818403 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 % _timestamp‚1.7843365964925575e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚303693870921216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4668987893248 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3620475733504 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32793731 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66112260»WæŠ2÷- _wandbruntime‚6757  _runtime‚6757’7Èëø‚ 0loss/FRt2/assignment_loss‚0.9630402514729837 /loss/FRt1/detection_loss‚0.5430204098917968 /loss/FBt2/detection_loss‚0.9911507026202255 $loss/total_loss‚9.10061959556932 % _timestamp‚1.7843365965544963e+09 _runtime‚6757.139620514 2loss/FBt2/assignment_loss‚0.049386183787308045 /loss/FBt1/detection_loss‚0.4950022797540961 trainer/global_step‚3849 _step‚316 0loss/FRt1/assignment_loss‚1.7634524327820775 2loss/SRqqt1/assignment_loss‚0.5052035188967542 2loss/SRqqt2/assignment_loss‚0.5217888097497737 2loss/FBt1/assignment_loss‚0.043115421759544095 /loss/FRt2/detection_loss‚1.7068031164699922 1loss/SRqqt1/detection_loss‚0.7902908315506162 1loss/SRqqt2/detection_loss‚0.7283656368341528 epoch‚9û<³¡ ù‡ 2loss/SRqqt1/assignment_loss‚0.5052035188967542 2loss/SRqqt2/assignment_loss‚0.5217888097497737 2loss/FBt1/assignment_loss‚0.043115421759544095 /loss/FRt2/detection_loss‚1.7068031164699922 1loss/SRqqt1/detection_loss‚0.7902908315506162 1loss/SRqqt2/detection_loss‚0.7283656368341528 epoch‚9 0loss/FRt2/assignment_loss‚0.9630402514729837 /loss/FRt1/detection_loss‚0.5430204098917968 /loss/FBt2/detection_loss‚0.9911507026202255 $loss/total_loss‚9.10061959556932 % _timestamp‚1.7843365965544963e+09 _runtime‚6757.139620514 2loss/FBt2/assignment_loss‚0.049386183787308045 /loss/FBt1/detection_loss‚0.4950022797540961 trainer/global_step‚3849 _step‚316 0loss/FRt1/assignment_loss‚1.7634524327820775¼¢x¼ì„ú‚(:‰ Ü¡ëÒÙ—˜È gpu.0.gpu‚86.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllo]¤¢WcatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚253.989' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7978 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FðhWêû‚(:ß Ü¡ëÒ±’ìÉ disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0" disk.dm-0.out‚44083.1103515625 disk.sda.out‚ 283.45703125! disk.nvme0n1.in‚ 8460.57421875$ disk./.usageGB‚589.5663681030273 disk.sda1.out‚0 disk.dm-1.out‚ 283.45703125 disk.drbd2165.out‚0% disk.nvme0n1.out‚44083.1103515625$ memory_percent‚91.67449474334717) proc.memory.availableMB‚ 1364.05078125* proc.memory.percent‚17.805862426757812 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 disk.drbd2165.in‚0 network.recv‚ 1.697236e+06 disk.sda3.out‚ 283.45703125 disk.dm-1.in‚ 0.47265625' disk.nvme0c0n1.out‚44083.1103515625 network.sent‚ 7.299455e+06) disk./.usagePercent‚39.56629364252599 disk.rbd1.out‚ 18.1640625 disk.sda.in‚ 0.47265625 disk.sda1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 8460.57421875 disk.dm-0.in‚ 8460.57421875 cpu‚25.522211067376404 proc.memory.rssMB‚ 2917.3125 proc.cpu.threads‚174ÁÃ.”ü‚(:‰ ë¡ëÒ€ä¦É gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚270.414' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ÒÚìý‚(:á ë¡ëÒ‚ÒÂÊ$ disk./.usageGB‚589.5309791564941 disk.sda.in‚ 0.47265625 proc.memory.rssMB‚ 2917.3125 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 network.sent‚ 7.364572e+06 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 284.30078125 disk.sda.out‚ 284.30078125# disk.nvme0c0n1.in‚ 8460.57421875( disk.nvme0c0n1.out‚44222.40087890625 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 8460.57421875& disk.nvme0n1.out‚44222.40087890625 disk.sda1.in‚0$ memory_percent‚91.67323112487793 cpu‚25.521899701723402 disk.rbd1.in‚0 network.recv‚ 1.702277e+06 disk.sda2.out‚0 disk.rbd1.out‚ 18.1640625# disk.dm-0.out‚44222.40087890625( proc.memory.availableMB‚ 1364.2578125 disk.rbd0.out‚0 disk.dm-0.in‚ 8460.57421875* proc.memory.percent‚17.805862426757812) disk./.usagePercent‚39.56391866063084 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 284.30078125 proc.cpu.threads‚17Ô2Ñ”þ‚(:‰ ú¡ëÒ«ù¬Ç gpu.0.gpu‚86.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚288.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.7936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨ êÿ‚(:ß ú¡ëÒ­Š´É( disk.nvme0c0n1.out‚44252.34814453125$ disk./.usageGB‚589.5315055847168$ memory_percent‚91.67344570159912* proc.memory.percent‚17.805862426757812 disk.dm-1.in‚ 0.47265625 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625 disk.nvme0n1.in‚ 8529.3984375& disk.nvme0n1.out‚44252.34814453125 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.47265625 proc.memory.rssMB‚ 2917.3125 cpu‚25.521530299562233) disk./.usagePercent‚39.56395398967057 disk.sda.out‚ 287.19921875 disk.drbd2165.in‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1364.22265625 network.sent‚ 7.367743e+06 network.recv‚ 1.705924e+06 disk.drbd2165.out‚0 disk.sda3.out‚ 287.19921875" disk.nvme0c0n1.in‚ 8529.3984375 disk.dm-1.out‚ 287.19921875 disk.rbd1.out‚ 18.2421875 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 8529.3984375# disk.dm-0.out‚44252.34814453125 disk.sda1.out‚0\èú²ž€‚(:“ ‰¢ëÒ¸Ç×È gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚295.687' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.13740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒÙé‚(:Þ ‰¢ëÒ¹¸öÈ disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 288.57421875* proc.memory.percent‚17.805862426757812( disk.nvme0c0n1.out‚44256.39794921875 disk.sda3.out‚ 288.57421875 disk.sda2.out‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚44256.39794921875 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 disk.dm-1.in‚ 0.47265625( proc.memory.availableMB‚ 1364.1953125 network.recv‚ 1.709569e+06 disk.drbd2165.in‚0 cpu‚25.521194859059154) disk./.usagePercent‚39.56394502940687$ disk./.usageGB‚589.5313720703125 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 8529.3984375 disk.rbd1.out‚ 18.2421875 disk.sda.in‚ 0.47265625 disk.sda.out‚ 288.57421875 disk.sda1.in‚0 disk.rbd1.in‚0$ memory_percent‚91.67361259460449 network.sent‚ 7.372608e+06 proc.memory.rssMB‚ 2917.3125 disk.dm-0.in‚ 8529.3984375# disk.dm-0.out‚44256.39794921875 disk.nvme0n1.in‚ 8529.3984375ß‚N‚‚(:… ˜¢ëÒªõ”É gpu.0.gpu‚97.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚293.8' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.76 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PG1?僂(:Ú ˜¢ëÒ£ÿºÊ$ disk./.usageGB‚589.5313758850098 disk.rbd1.in‚0! disk.nvme0n1.in‚ 8530.36328125 disk.sda1.out‚0 cpu‚25.520916956961543 disk.sda2.in‚0 disk.sda.in‚ 0.47265625 network.recv‚ 1.713036e+06 disk.sda.out‚ 289.109375 network.sent‚ 7.37762e+06* disk./.usagePercent‚39.563945285414405 disk.sda2.out‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 8530.36328125 disk.sda3.out‚ 289.109375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 289.109375 disk.rbd1.out‚ 18.3046875 disk.dm-0.in‚ 8530.36328125 disk.sda3.in‚ 0.47265625$ memory_percent‚91.66955947875977 proc.memory.rssMB‚ 2917.3125 disk.drbd2165.in‚0# disk.dm-0.out‚44270.20556640625 disk.sda1.in‚0' proc.memory.availableMB‚ 1364.859375 disk.dm-1.in‚ 0.47265625& disk.nvme0n1.out‚44270.20556640625* proc.memory.percent‚17.805862426757812 proc.cpu.threads‚17( disk.nvme0c0n1.out‚44270.20556640625Sÿž—2„- _wandbruntime‚6839  _runtime‚6839@‡ý˜Æ…À & lr-AdamW/pg1‚0.0004455182174775849 & lr-AdamW/pg2‚0.0004455182174775849 trainer/global_step‚3899 $ _timestamp‚1.784336678519157e+09 _runtime‚6839.104212731 _step‚317Tl„<ˆŠtrainer/global_step‚3899 $ _timestamp‚1.784336678519157e+09 _runtime‚6839.104212731 _step‚317 & lr-AdamW/pg1‚0.0004455182174775849 & lr-AdamW/pg2‚0.0004455182174775849½'ˆÝ2‡- _wandbruntime‚6839  _runtime‚6839¥Ù"?ÿ'ˆùO VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1121916855468724 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _step‚318 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3658251784884 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1118742156419072 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33745782 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11076371 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3664521231872 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1118258603683840 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1122406677650944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _runtime‚6839.105835163 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33195819 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33195819 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21078780 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11076614 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4727913398784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33735809 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307549664932352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843366785204341e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33735809 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3665866621440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21078532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10002166 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 trainer/global_step‚3899 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1118742182592512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10002161 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1122408049213952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1122408049213952 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1122406677650944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0«ñˆ(‰þO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33195819 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21078780 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11076614 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4727913398784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33735809 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307549664932352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843366785204341e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33735809 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3665866621440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21078532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10002166 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 trainer/global_step‚3899 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1118742182592512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10002161 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1122408049213952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1122408049213952 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1122406677650944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1121916855468724 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _step‚318 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3658251784884 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1118742156419072 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33745782 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11076371 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3664521231872 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1118258603683840 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1122406677650944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _runtime‚6839.105835163 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33195819 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248¾6•Õ8žŠ‚(:“ §¢ëÒ£ž½Ç gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚276.392' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.27839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+úšÛ‹‚(:Ð §¢ëÒÖñÉ disk.dm-0.out‚44278.69921875 disk.dm-1.in‚ 0.47265625$ memory_percent‚91.66829586029053 network.recv‚ 1.715716e+06% disk.nvme0c0n1.out‚44278.69921875 disk.dm-1.out‚ 289.640625 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0 proc.memory.rssMB‚ 2917.3125* proc.memory.percent‚17.805862426757812 cpu‚25.520547907411707) disk./.usagePercent‚39.56395526970824# disk.nvme0n1.out‚44278.69921875 disk.sda3.out‚ 289.640625 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.sent‚ 7.381684e+06" disk.nvme0c0n1.in‚ 8531.1953125 disk.nvme0n1.in‚ 8531.1953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 8531.1953125 disk.sda.out‚ 289.640625$ disk./.usageGB‚589.5315246582031 disk.rbd0.in‚0) proc.memory.availableMB‚ 1365.06640625 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 18.3046875‚׿‘2Œ- _wandbruntime‚6840  _runtime‚6840¹‚¥Ý ƒN EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3666791664128 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3665446267392 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33744287 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1119020157149184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4729152047616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStaa¼Ï ,tsMonitor.on_train_batch_end/segment.all.current‚1995 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1119020183322624 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33204025 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21083826 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚302899237941248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10004704 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3666791664128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66958305 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1122195660713146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33204025 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66958305 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1118536485462016 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33204031 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33754274 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66948312 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚3899 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3665446267392 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _runtime‚6840.523910442 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1122686974986752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33754274 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33744287 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11079122 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚319 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33204031 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21084065 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚307628758080000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1122685603416576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1118536510889984ížw.'ŽˆN YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚302899606032384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33744287 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1119020157149184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4729152047616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1119020183322624 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33204025 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21083826 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚302899237941248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10004704 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3666791664128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66958305 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1122195660713146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33204025 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66958305 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1118536485462016 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33204031 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33754274 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66948312 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚3899 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3665446267392 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _runtime‚6840.523910442 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1122686974986752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33754274 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33744287 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11079122 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚319 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33204031 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21084065 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚307628758080000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1122685603416576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1118536510889984 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3666791664128 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3665446267392 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21344449 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4788360011264 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1132642219098112 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1132152718020608 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1136354401650176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚306696440008704 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3705833498314 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4788327571968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3704488897992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67777659 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33606125 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11214897 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21344208 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1132152692592640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34161561 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34161561 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1136353030087168 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34171534 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33606125 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33606119 JADeviceStatsMonitor.on’úfÍV _train_batch_start/active.small_pool.current‚9973 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1136354401650176 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1136353030087168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚311484800019968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1132642219098112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311484399489536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3710837162496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3712182552064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚306696071917568 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _runtime‚6922.524841915 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10129311 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33606119 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11215133 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1132642192924672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1132642192924672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 % _timestamp‚1.7843367619394264e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008;š#(¡þO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1132642219098112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311484399489536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3710837162496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3712182552064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚306696071917568 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _runtime‚6922.524841915 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10129311 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33606119 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11215133 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1132642192924672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1132642192924672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 % _timestamp‚1.7843367619394264e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34171534 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21344449 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4788360011264 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1132642219098112 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1132152718020608 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1136354401650176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚306696440008704 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3705833498314 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4788327571968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3704488897992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67777659 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33606125 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11214897 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21344208 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1132152692592640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34161561 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34161561 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1136353030087168 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34171534 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33606125 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33606119 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1136354401650176 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1136353030087168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚311484800019968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ÂZIÏ2¢- _wandbruntime‚6923  _runtime‚6923ë%g '£ƒN VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10131859 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67784364 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _runtime‚6923.971528049 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34170039 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4789566990848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33614331 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67784364 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚311563092883968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10131854 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1132430599798784 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3705412364238 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1132920193654784 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1132920193654784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4789534558720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34180026 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1136137356763456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11217618 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1136633327422976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33614325 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33614325 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34180026 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3706756964672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚311563493407232 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67794357 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1132920219828224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 % _timestamp‚1.7843367633860104e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1132920219828224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67794357 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21349472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3949 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1136631955852800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21349709 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3713107594752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚306773558325248 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1136135986735054 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1132430574370816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3711762198016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33614331 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097ZìÖH#¤ˆN TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34180026 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1136137356763456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11217618 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1136633327422976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33614325 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33614325 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34180026 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3706756964672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚311563493407232 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67794357 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1132920219828224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 % _timestamp‚1.7843367633860104e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1132920219828224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67794357 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21349472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3949 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1136631955852800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21349709 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3713107594752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚306773558325248 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1136135986735054 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1132430574370816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3711762198016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33614331 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10131859 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67784364 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _runtime‚6923.971528049 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34170039 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4789566990848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33614331 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67784364 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚311563092883968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.´60>Æfreed‚10131854 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1132430599798784 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3705412364238 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1132920193654784 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1132920193654784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4789534558720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440Ãèª2¥- _wandbruntime‚6924  _runtime‚6924(»zµ ¦ƒ 0loss/FRt2/assignment_loss‚0.8448556366005564 2loss/SRqqt1/assignment_loss‚0.4761232082847416 _runtime‚6924.023499631 _step‚324 /loss/FRt1/detection_loss‚0.5356626660071196 1loss/SRqqt1/detection_loss‚0.7029265685119015 trainer/global_step‚3949 % _timestamp‚1.7843367634383326e+09 1loss/FBt1/assignment_loss‚0.03489024973076007 /loss/FRt2/detection_loss‚1.7902117238902464 1loss/SRqqt2/detection_loss‚0.7218629671460391 /loss/FBt2/detection_loss‚0.8210177882700354 epoch‚9 0loss/FRt1/assignment_loss‚1.7568512595267523 2loss/SRqqt2/assignment_loss‚0.5368367122846864 2loss/FBt2/assignment_loss‚0.009393550559387813 0loss/FBt1/detection_loss‚0.42009628556665407 %loss/total_loss‚8.650728616378881½€§§ˆ _runtime‚6924.023499631 _step‚324 /loss/FRt1/detection_loss‚0.5356626660071196 1loss/SRqqt1/detection_loss‚0.7029265685119015 trainer/global_step‚3949 % _timestamp‚1.7843367634383326e+09 1loss/FBt1/assignment_loss‚0.03489024973076007 /loss/FRt2/detection_loss‚1.7902117238902464 1loss/SRqqt2/detection_loss‚0.7218629671460391 /loss/FBt2/detection_loss‚0.8210177882700354 epoch‚9 0loss/FRt1/assignment_loss‚1.7568512595267523 2loss/SRqqt2/assignment_loss‚0.5368367122846864 2loss/FBt2/assignment_loss‚0.009393550559387813 0loss/FBt1/detection_loss‚0.42009628556665407 %loss/total_loss‚8.650728616378881 0loss/FRt2/assignment_loss‚0.8448556366005564 2loss/SRqqt1/assignment_loss‚0.4761232082847416ÄÓ¶uU”¨‚(:‰ £ëÒš’ÅÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$8fúë©‚(:à £ëÒûž™É disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 300.17578125 disk.sda.in‚ 0.47265625 proc.memory.rssMB‚ 2917.3125 network.recv‚ 1.738813e+06 disk.rbd1.out‚ 18.54296875* disk./.usagePercent‚39.360952559387194! disk.nvme0n1.in‚ 8657.30078125 disk.rbd0.in‚0 disk.dm-1.out‚ 300.17578125$ memory_percent‚91.67475700378418 proc.cpu.threads‚17 disk.dm-0.in‚ 8657.30078125 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0 disk.sda.out‚ 300.17578125# disk.nvme0c0n1.in‚ 8657.30078125' disk.nvme0c0n1.out‚44506.6708984375$ disk./.usageGB‚586.5066375732422 disk.sda1.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0( proc.memory.availableMB‚ 1364.0078125* proc.memory.percent‚17.805862426757812 network.sent‚ 7.506729e+06" disk.dm-0.out‚44506.6708984375% disk.nvme0n1.out‚44506.6708984375 disk.sda2.in‚0 disk.drbd2165.out‚0 cpu‚25.518705747619144…ÀµT”ª‚(:‰ £ëÒ׸ÀÇ gpu.0.gpu‚97.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚264.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03˦fâ«‚(:× £ëÒ Í‡Ë network.sent‚ 7.569869e+06) disk./.usagePercent‚39.36095230337966 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.memory.rssMB‚ 2917.3125 network.recv‚ 1.743306e+06 disk.dm-1.out‚ 300.3359375 disk.rbd1.in‚0 disk.rbd1.out‚ 18.60546875" disk.dm-0.out‚44517.0029296875 disk.sda.out‚ 300.3359375" disk.nvme0c0n1.in‚ 8658.1328125' disk.nvme0c0n1.out‚44517.0029296875 disk.dm-0.in‚ 8658.1328125 disk.sda1.in‚0 disk.sda.in‚ 0.47265625% disk.nvme0n1.out‚44517.0029296875 disk.sda3.out‚ 300.3359375 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 cpu‚25.5184002836802 disk.sda1.out‚0 disk.dm-1.in‚ 0.47265625$ memory_percent‚91.67370796203613( proc.memory.availableMB‚ 1364.1796875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 8658.1328125$ disk./.usageGB‚586.5066337585449 disk.sda3.in‚ 0.47265625* proc.memory.percent‚17.805862426757812i-û„”¬‚(:‰ Ÿ£ëÒñÁ¼Ç gpu.0.gpu‚83.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚264.538' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9076 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿¢ã­‚(:Ø Ÿ£ëÒø¯½É disk.rbd0.in‚0 disk.sda.out‚ 300.78125 disk.drbd2165.out‚0$ memory_percent‚91.67370796203613# disk.nvme0c0n1.in‚ 8659.09765625 disk.dm-0.in‚ 8659.09765625& disk.nvme0n1.out‚44524.87451171875 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 300.78125 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 300.78125 disk.rbd1.in‚0 disk.drbd2165.in‚0* proc.memory.percent‚17.805862426757812 network.recv‚ 1.746864e+06( disk.nvme0c0n1.out‚44524.87451171875! disk.nvme0n1.in‚ 8659.09765625 disk.rbd1.out‚ 18.60546875$ disk./.usageGB‚586.5068168640137 disk.sda2.out‚0) disk./.usagePercent‚39.36096459174131 disk.rbd0.out‚0 disk.sda2.in‚0# disk.dm-0.out‚44524.87451171875 disk.sda1.in‚0 proc.memory.rssMB‚ 2917.3125 proc.cpu.threads‚17 network.sent‚ 7.574774e+06 disk.sda.in‚ 0.47265625( proc.memory.availableMB‚ 1364.1796875 cpu‚25.51808641648108 disk.sda1.out‚0é/d”®‚(:‰ ®£ëÒµƒ¥Ç gpu.0.gpu‚86.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ?RÒ߯‚(:Ô ®£ëÒó£ÇÉ disk.sda.out‚ 301.8515625 disk.drbd2165.out‚0! disk.dm-0.out‚44534.052734375 network.sent‚ 7.579778e+06$ disk./.usageGB‚586.5035514831543 disk.sda3.in‚ 0.47265625 proc.memory.rssMB‚ 2917.3125 cpu‚25.517761349209756 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625" disk.nvme0c0n1.in‚ 8751.4453125' disk./.usagePercent‚39.360745449292 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625 disk.sda1.out‚0 disk.nvme0n1.in‚ 8751.4453125$ disk.nvme0n1.out‚44534.052734375 disk.rbd0.in‚0 disk.dm-0.in‚ 8751.4453125 disk.sda3.out‚ 301.8515625 disk.dm-1.out‚ 301.8515625 disk.drbd2165.in‚0( proc.memory.availableMB‚ 1364.1640625* proc.memory.percent‚17.805862426757812 proc.cpu.threads‚17& disk.nvme0c0n1.out‚44534.052734375$ memory_percent‚91.67380332946777 network.recv‚ 1.750119e+06 disk.rbd1.out‚ 18.68359375«·Î“°‚(:ˆ ½£ëÒ嫵Ç gpu.0.gpu‚97.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùlÓÝܱ‚(:Ñ ½£ëÒÝœµÉ network.recv‚ 1.753721e+06 disk.sda.out‚ 303.71875# disk.nvme0c0n1.in‚ 8752.27734375 disk.dm-0.in‚ 8752.27734375 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 303.71875 cpu‚25.517474981117065# disk.nvme0n1.out‚44546.94921875 disk.rbd1.in‚0 disk.rbd1.out‚ 18.68359375 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17$ memory_percent‚91.67523384094238 network.sent‚ 7.584841e+06* disk./.usagePercent‚39.360819179461856% disk.nvme0c0n1.out‚44546.94921875 disk.sda1.in‚0 proc.memory.rssMB‚ 2917.3125* proc.memory.percent‚17.805862426757812 disk.rbd0.out‚0 disk.sda2.out‚0$ disk./.usageGB‚586.5046501159668 disk.dm-0.out‚44546.94921875 disk.dm-1.out‚ 303.71875( proc.memory.availableMB‚ 1363.9296875! disk.nvme0n1.in‚ 8752.27734375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 0.47265625!Lû”²‚(:‰ Ì£ëÒÍä¸Ç gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚241.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.3292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0㘧ë糂(:Ü Ì£ëÒ̤­Ê cpu‚25.51714614923376 disk.dm-1.out‚ 305.2265625$ memory_percent‚91.67373180389404 disk.sda1.out‚0 disk.sda2.in‚0 proc.memory.rssMB‚ 2917.3125$ disk./.usageGB‚586.5029830932617 disk.rbd1.out‚ 18.80859375 disk.sda.out‚ 305.2265625 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 8753.2421875 network.recv‚ 1.757043e+06 disk.dm-1.in‚ 0.47265625 disk.nvme0n1.in‚ 8753.2421875 disk.rbd0.in‚0) disk./.usagePercent‚39.36070730416939 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 305.2265625 disk.sda2.out‚0) proc.memory.availableMB‚ 1364.17578125* proc.memory.percent‚17.805862426757812 network.sent‚ 7.589954e+06( disk.nvme0c0n1.out‚44560.60107421875 disk.dm-0.in‚ 8753.2421875 disk.rbd1.in‚0# disk.dm-0.out‚44560.60107421875& disk.nvme0n1.out‚44560.60107421875 proc.cpu.threads‚17 disk.sda.in‚ 0.47265625 disk.rbd0.out‚0|Çû2´- _wandbruntime‚7005  _runtime‚7005$h¯ÇµÁ _step‚325 & lr-AdamW/pg1‚0.0004434353967316584 & lr-AdamW/pg2‚0.0004434353967316584 trainer/global_step‚3999 % _timestamp‚1.7843368453703985e+09 _runtime‚7005.955536494uÌ¶Æ trainer/global_step‚3999 % _timestamp‚1.7843368453703985e+09 _runtime‚7005.955536494 _step‚325 & lr-AdamW/pg1‚0.0004434353967316584 & lr-AdamW/pg2‚0.0004434353967316584Å°š±2·- _wandbruntime‚7005  _runtime‚7005ÚæÙÿ'¸ùO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1146542229430272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3752070611422 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1146542229430272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11353277 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34016425 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315419154448896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3750726011100 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1150299382523392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _runtime‚7005.957452621 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚3999 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68613711 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10256466 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68603732 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10256461 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34587313 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1150300754086400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21609989 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4848359519744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34016419 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34597286 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1146046781501440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1146542255603712 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10256466 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68603732 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10256461 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34587313 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1150300754086400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21609989 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4848359519744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34016419 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34597286 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1146046781501440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1146542255603712 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1150299382523392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _runtime‚7005.957452621 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚3999 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68613711 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1150578308289024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3759423525376 TEDeviceStatsMonit’O=Yµ$or.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3752994077780 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚315497434279424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1146324663279616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 % _timestamp‚1.7843368467865882e+09 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34595791 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68630409 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3758078128640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10259009 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1150076312756962 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34595791 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34024631 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1150077682785364 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3751649477346 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68630409 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34024625 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3759423525376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68620416 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10259004 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚315497836899840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310647878709248 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68620416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34024625 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1146324688707584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 trainer/global_step‚3999 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4849590099456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1150579679859200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34605778 _step‚327 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1146820230160384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11356018 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11356254 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1146820256333824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1146820230160384 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3758078128640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4849555570176 _runtime‚7007.371972017 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1150579679859200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21615022 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119Ñ-Ž'¼ˆN M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310647878709248 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68620416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34024625 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1146324688707584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 trainer/global_step‚3999 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4849590099456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1150579679859200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34605778 _step‚327 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1146820230160384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11356018 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11356254 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1146820256333824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1146820230160384 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3758078128640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4849555570176 _runtime‚7007.371972017 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1150579679859200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21615022 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1150578308289024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3759423525376 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3752994077780 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚315497434279424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1146324663279616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 % _timestamp‚1.7843368467865882e+09 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34595791 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68630409 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3758078128640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10259009 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1150076312756962 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34595791 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34024631 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1150077682785364 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3751649477346 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68630409 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34024625 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3759423525376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68620416 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10259004 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚315497836899840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216ÇÑ’|ñ2½- _wandbruntime‚7007  _runtime‚7007ßÉs ¾ý 2loss/SRqqt1/assignment_loss‚0.5251304660255862 2loss/SRqqt2/assignment_loss‚0.5630912013453464 1loss/SRqqt1/detection_loss‚0.7720921141729203 epoch‚9 1loss/SRqqt2/detection_loss‚0.7211984188602485 0loss/FBt1/detection_loss‚0.38525564374546684 _step‚328 /loss/FRt1/assignment_loss‚1.650490130680697 0loss/FRt2/assignment_loss‚0.9408280098986683 -loss/FRt2/detection_loss‚1.66550870405253 $loss/total_loss‚9.98054978283628 trainer/global_step‚3999 % _timestamp‚1.7843368468543916e+09 _runtime‚7007.439489134 1loss/FBt1/assignment_loss‚0.08662101148378966 1loss/FBt2/assignment_loss‚0.02764891651493361 /loss/FRt1/detection_loss‚0.5397952640311638 .loss/FBt2/detection_loss‚2.102889902024929$"›Ü¿‚ 0loss/FRt2/assignment_loss‚0.9408280098986683 -loss/FRt2/detection_loss‚1.66550870405253 $loss/total_loss‚9.98054978283628 trainer/global_step‚3999 % _timestamp‚1.7843368468543916e+09 _runtime‚7007.439489134 1loss/FBt1/assignment_loss‚0.08662101148378966 1loss/FBt2/assignment_loss‚0.02764891651493361 /loss/FRt1/detection_loss‚0.5397952640311638 .loss/FBt2/detection_loss‚2.102889902024929 2loss/SRqqt1/assignment_loss‚0.5251304660255862 2loss/SRqqt2/assignment_loss‚0.5630912013453464 1loss/SRqqt1/detection_loss‚0.7720921141729203 epoch‚9 1loss/SRqqt2/detection_loss‚0.7211984188602485 0loss/FBt1/detection_loss‚0.38525564374546684 _step‚328 /loss/FRt1/assignment_loss‚1.650490130680697Èn. À‚(:’ Û£ëÒ¿Å…Ç gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09—«ãáÁ‚(:Ö Û£ëÒÙÒ¯É) disk./.usagePercent‚39.36073930511117 disk.rbd1.in‚0 disk.drbd2165.in‚0$ disk./.usageGB‚586.5034599304199 disk.nvme0n1.in‚ 8753.2421875 disk.sda3.out‚ 305.8203125 disk.rbd1.out‚ 18.80859375$ memory_percent‚91.67978763580322 proc.cpu.threads‚17! disk.dm-0.out‚44564.123046875 disk.sda2.out‚0& disk.nvme0c0n1.out‚44564.123046875" proc.memory.rssMB‚ 2918.4765625 disk.dm-0.in‚ 8753.2421875 disk.sda3.in‚ 0.47265625" disk.nvme0c0n1.in‚ 8753.2421875) proc.memory.percent‚17.81296730041504 cpu‚25.521197879047 disk.rbd0.in‚0 disk.sda.out‚ 305.8203125 network.sent‚ 7.594845e+06 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 305.8203125 disk.sda1.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚44564.123046875 disk.dm-1.in‚ 0.47265625 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1363.18359375 network.recv‚ 1.760488e+06'd&"žÂ‚(:“ ê£ëÒ°·ÎÇ gpu.0.gpu‚99.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚329.321' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.86420000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r¹èÂ(:Ý ê£ëÒ·‰ÊÊ network.recv‚ 1.764855e+06* disk./.usagePercent‚39.362949674161804 disk.nvme0n1.in‚ 9005.890625 disk.rbd1.in‚0 disk.rbd1.out‚ 18.88671875 disk.sda.out‚ 307.9140625 disk.sda3.out‚ 307.9140625 disk.rbd0.in‚0 disk.sda3.in‚ 0.47265625 disk.sda2.out‚0 disk.dm-0.in‚ 9005.890625 disk.sda1.out‚0( disk.nvme0c0n1.out‚44618.28564453125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.sent‚ 7.660112e+06 disk.rbd0.out‚0 disk.sda.in‚ 0.47265625 cpu‚25.52624280300394& disk.nvme0n1.out‚44618.28564453125 disk.sda1.in‚0 disk.dm-1.out‚ 307.9140625# disk.dm-0.out‚44618.28564453125$ disk./.usageGB‚586.5363960266113! disk.nvme0c0n1.in‚ 9005.890625# memory_percent‚91.7499303817749) proc.memory.availableMB‚ 1351.69140625# proc.memory.rssMB‚ 2930.06640625* proc.memory.percent‚17.883706092834473­íÖ2Ä- _wandbruntime‚7035  _runtime‚7035ÉFr„}"Å¡{ 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9952172636985779 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.37962964177131653 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5766657590866089 4 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9876748919487 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9929001331329346 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6456795930862427 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 /particle/accuracy_3_of_3‚0.9994933605194092 jet/accuracy_4_of_5‚0 _step‚329 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.992289662361145 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2154185175895691 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05082620307803154 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8863970637321472 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5131843686103821 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6881173253059387 ,jet/accuracy_4_of_4‚0.013899499550461769 (particle/accuracy‚0.8473153114318848 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.283573180437088 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2717004120349884 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7466288805007935 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14586347341537476 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_1_of_1‚0.31801724433898926 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5873715281486511 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6309415102005005 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2825012803077698 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5364962220191956 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5450944900512695 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6794621348381042 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 /particle/accuracy_3_of_4‚0.9968253374099731 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6380208134651184 jet/accuracy_5_of_5‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6292851567268372 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9753497242927551 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5364962220191956 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6816328763961792 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2863720953464508 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49895617365837097 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9181712865829468 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6492681503295898 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28523916006088257 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2950553596019745 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.282438725233078 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04344802722334862 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14586347341537476 *jet/accuracy_2_of_2‚0.1333530694246292 particle/accuracy_1_of_2‚1 jet/accuracy_1_of_5‚1 trainer/global_step‚3999 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997359037399292 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.07961837947368622 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8501385450363159 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9833317995071411 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09864910691976547 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08447076380252838 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5117855072021484 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.38461539149284363 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.7685185074806213 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8212395906448364 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6213701963424683 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7685185074806213 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 epoch‚9 5 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.97822505235672 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.996832549571991 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2851451337337494 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_1_of_3‚0.9212728142738342 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 4 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2489003688097 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977155327796936 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15487025678157806 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.533842921257019 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7466288805007935 particle/accuracy_2_of_4‚1 +particle/sensitivity‚0.6342869997024536 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.603123664855957 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4768518805503845 *jet/accuracy_2_of_3‚0.5078681707382202 7validation_average_jet_accuracy‚0.36025357246398926 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.807692289352417 particle/accuracy_4_of_5‚1 _runtime‚7035.671303229 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5810831189155579 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.37962964177131653 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3203698992729187 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 5 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.54708331823349 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29610109329223633 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_3‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7512850165367126 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5762190818786621 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15430991351604462 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5392371416091919 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2765188217163086 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.26097792387008667 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 &particle/f_score‚0.640808641910553 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5785697102546692 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2821232080459595 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5768882036209106 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.30509963631629944 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7685185074806213 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6390421986579895 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5875851511955261 particle/accuracy_3_of_5‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 *jet/accuracy_2_of_4‚0.7771109342575073 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2734954059123993 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4083150327205658 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5880365967750549 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9822333455085754 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1567167490720749 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.996832549571991 A&Purity/1FRt2SRqqt1FBt/event_propor$ÞàP*tion‚0.00027901786961592734 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1458333432674408 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.26440298557281494 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8617536425590515 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32413363456726074 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8652304410934448 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9915086030960083 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9827397465705872 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.13900022208690643 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5746294260025024 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2851451337337494 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6340829133987427 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9927902221679688 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7222222089767456 /particle/accuracy_4_of_4‚0.9567582011222839 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6268264055252075 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15794116258621216 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.26172178983688354 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32156050205230713 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8548375368118286 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_1‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.38461539149284363 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6113880276679993 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.282885879278183 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_5_of_5‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1363763064146042 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666567325592 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.339305579662323 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942795634269714 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5791104435920715 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22379906475543976 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6666424870491028 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3470749855041504 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5046756267547607 particle/accuracy_1_of_3‚1 % _timestamp‚1.7843368750855289e+09 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 particle/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9263070225715637 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3142411708831787 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6446605324745178 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8660250306129456 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4535999298095703 *jet/accuracy_3_of_3‚0.0692138597369194 *jet/accuracy_1_of_4‚0.9692099690437317 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6226586699485779 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.26967528462409973 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8640967607498169 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.3386088013648987 )loss/val_total_loss‚8.799607111996218 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7261151671409607 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9775032997131348 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6340668201446533 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6290333271026611 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *particle/specificity‚0.905571460723877 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9896320104598999 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9940537214279175 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9742063879966736 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5714701414108276 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9965658783912659 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03946730121970177 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9962283372879028 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 *jet/accuracy_1_of_2‚0.7846589088439941 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.27149486541748047 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.543857753276825 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04984518513083458 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6334958672523499 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2941552400588989 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1$»¬=Ʀ{ 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997359037399292 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.07961837947368622 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8501385450363159 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9833317995071411 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09864910691976547 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08447076380252838 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5117855072021484 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.38461539149284363 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.7685185074806213 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8212395906448364 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6213701963424683 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7685185074806213 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 epoch‚9 5 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.97822505235672 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.996832549571991 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2851451337337494 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_1_of_3‚0.9212728142738342 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 4 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2489003688097 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977155327796936 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15487025678157806 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.533842921257019 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7466288805007935 particle/accuracy_2_of_4‚1 +particle/sensitivity‚0.6342869997024536 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.603123664855957 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4768518805503845 *jet/accuracy_2_of_3‚0.5078681707382202 7validation_average_jet_accuracy‚0.36025357246398926 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.807692289352417 particle/accuracy_4_of_5‚1 _runtime‚7035.671303229 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5810831189155579 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.37962964177131653 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3203698992729187 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 5 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.54708331823349 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29610109329223633 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_3‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7512850165367126 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5762190818786621 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15430991351604462 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5392371416091919 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2765188217163086 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.26097792387008667 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 &particle/f_score‚0.640808641910553 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5785697102546692 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2821232080459595 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5768882036209106 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.30509963631629944 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7685185074806213 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6390421986579895 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5875851511955261 particle/accuracy_3_of_5‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 *jet/accuracy_2_of_4‚0.7771109342575073 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2734954059123993 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4083150327205658 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5880365967750549 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9822333455085754 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1567167490720749 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.996832549571991 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1458333432674408 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.26440298557281494 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8617536425590515 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32413363456726074 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8652304410934448 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9915086030960083 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9827397465705872 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.13900022208690643 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5746294260025024 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2851451337337494 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6340829133987427 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9927902221679688 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7222222089767456 /particle/accuracy_4_of_4‚0.9567582011222839 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6268264055252075 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15794116258621216 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.26172178983688354 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32156050205230713 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8548375368118286 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_1‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.38461539149284363 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6113880276679993 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.282885879278183 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_5_of_5‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1363763064146042 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666567325592 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.339305579662323 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942795634269714 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5791104435920715 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22379906475543976 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6666424870491028 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3470749855041504 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5046756267547607 particle/accuracy_1_of_3‚1 % _timestamp‚1.7843368750855289e+09 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 particle/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9263070225715637 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3142411708831787 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6446605324745178 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8660250306129456 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4535999298095703 *jet/accuracy_3_of_3‚0.0692138597369194 *jet/accuracy_1_of_4‚0.9692099690437317 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6226586699485779 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.26967528462409973 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8640967607498169 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.3386088013648987 )loss/val_total_loss‚8.799607111996218 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7261151671409607 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9775032997131348 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6340668201446533 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6290333271026611 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *particle/specificity‚0.905571460723877 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9896320104598999 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9940537214279175 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9742063879966736 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5714701414108276 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9965658783912659 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03946730121970177 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9962283372879028 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 *jet/accuracy_1_of_2‚0.7846589088439941 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.27149486541748047 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.543857753276825 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04984518513083458 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6334958672523499 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2941552400588989 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9952172636985779 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.37962964177131653 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5766657590866089 4 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9876748919487 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9929001331329346 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6456795930862427 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 /particle/accuracy_3_of_3‚0.9994933605194092 jet/accuracy_4_of_5‚0 _step‚329 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.992289662361145 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2154185175895691 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05082620307803154 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8863970637321472 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5131843686103821 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6881173253059387 ,jet/accuracy_4_of_4‚0.013899499550461769 (particle/accuracy‚0.8473153114318848 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.283573180437088 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2717004120349884 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7466288805007935 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14586347341537476 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_1_of_1‚0.31801724433898926 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5873715281486511 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6309415102005005 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2825012803077698 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5364962220191956 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5450944900512695 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6794621348381042 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 /particle/accuracy_3_of_4‚0.9968253374099731 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6380208134651184 jet/accuracy_5_of_5‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6292851567268372 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9753497242927551 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5364962220191956 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6816328763961792 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2863720953464508 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49895617365837097 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9181712865829468 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6492681503295898 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28523916006088257 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2950553596019745 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.282438725233078 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04344802722334862 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14586347341537476 *jet/accuracy_2_of_2‚0.1333530694246292 particle/accuracy_1_of_2‚1 jet/accuracy_1_of_5‚1 trainer/global_step‚3999 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935É>Åj‡ŸÇ‚(:” ù£ëÒÐî®È gpu.0.gpu‚93.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é‰qhìÈ‚(:á ù£ëÒ¸ôÉ disk.sda1.out‚0( disk.nvme0c0n1.out‚46597.87548828125* proc.memory.percent‚17.884039878845215 disk.rbd1.in‚0' proc.memory.availableMB‚ 1338.203125# proc.memory.rssMB‚ 2930.12109375 cpu‚25.524142703266502 disk.sda3.out‚ 309.0546875 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 9007.01171875# disk.dm-0.out‚46597.87548828125 disk.rbd0.out‚0& disk.nvme0n1.out‚46597.87548828125 disk.sda1.in‚0 network.sent‚ 7.663177e+06 disk.sda.out‚ 309.0546875 disk.sda2.out‚0 network.recv‚ 1.768687e+06( disk./.usagePercent‚39.3629855152166$ disk./.usageGB‚586.5369300842285 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 309.0546875 disk.rbd1.out‚ 18.88671875 disk.sda.in‚ 0.47265625! disk.nvme0n1.in‚ 9007.01171875 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 9007.01171875$ memory_percent‚91.83225631713867 proc.cpu.threads‚17oæìÓŸÉ‚(:” ˆ¤ëÒþóÉÇ gpu.0.gpu‚56.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›(?WïÊ‚(:ä ˆ¤ëÒµûùÉ" disk.dm-0.out‚46603.7529296875 disk.sda1.in‚0 disk.sda.in‚ 0.47265625 disk.dm-0.in‚ 9007.01171875 disk.sda1.out‚0# proc.memory.rssMB‚ 2930.12109375 disk.drbd2165.out‚0 network.sent‚ 7.720772e+06 disk.sda2.in‚0 disk.rbd1.out‚ 18.94921875 disk.drbd2165.in‚0 disk.sda3.in‚ 0.47265625* proc.memory.percent‚17.884039878845215 cpu‚25.523738653592304 disk.dm-1.out‚ 309.45703125 disk.sda.out‚ 309.45703125! disk.nvme0n1.in‚ 9007.01171875% disk.nvme0n1.out‚46603.7529296875 network.recv‚ 1.773262e+06 disk.rbd1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.36461756324738 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625# disk.nvme0c0n1.in‚ 9007.01171875' disk.nvme0c0n1.out‚46603.7529296875$ memory_percent‚91.83337688446045) proc.memory.availableMB‚ 1338.01953125 disk.sda3.out‚ 309.45703125 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚586.5612487792969”h*rŸË‚(:” —¤ëÒ¡ÌËÇ gpu.0.gpu‚32.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çíjÜêÌ‚(:ß —¤ëÒÊâ·Ê& disk.nvme0n1.out‚46610.77197265625 disk.rbd0.in‚0 disk.rbd1.in‚0 network.recv‚ 1.776898e+06$ disk./.usageGB‚586.5586471557617 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.47265625) proc.memory.availableMB‚ 1338.02734375 disk.drbd2165.out‚0 disk.dm-0.in‚ 9007.84375 disk.nvme0n1.in‚ 9007.84375 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 18.94921875 cpu‚25.523368362554162 proc.cpu.threads‚17 disk.sda.out‚ 309.94921875( disk.nvme0c0n1.out‚46610.77197265625 disk.dm-1.out‚ 309.94921875 network.sent‚ 7.725758e+06 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 309.94921875$ memory_percent‚91.83332920074463* disk./.usagePercent‚39.364442966109024 disk.rbd0.out‚0# proc.memory.rssMB‚ 2930.12109375# disk.dm-0.out‚46610.77197265625 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 9007.84375 disk.sda2.out‚0* proc.memory.percent‚17.884039878845215 disk.sda.in‚ 0.47265625m‡ÿ:žÍ‚(:“ ¦¤ëÒœ‡ÙÈ gpu.0.gpu‚23.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷|@ê΂(:ß ¦¤ëÒ½«ˆÊ" disk.dm-0.out‚46705.2607421875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0$ disk./.usageGB‚587.3066139221191 disk.rbd1.in‚0 disk.sda.out‚ 310.17578125% disk.nvme0n1.out‚46705.2607421875$ memory_percent‚91.83480739593506 proc.cpu.threads‚17 network.sent‚ 7.73089e+06) disk./.usagePercent‚39.41463964338519 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2930.12109375 disk.sda3.in‚ 0.47265625 disk.dm-1.out‚ 310.17578125* proc.memory.percent‚17.884039878845215 cpu‚25.52307603161827 disk.sda1.out‚0) proc.memory.availableMB‚ 1337.78515625 disk.dm-0.in‚ 9015.4453125 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 network.recv‚ 1.780703e+06 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 19.02734375 disk.sda.in‚ 0.47265625" disk.nvme0c0n1.in‚ 9015.4453125 disk.nvme0n1.in‚ 9015.4453125 disk.sda3.out‚ 310.17578125' disk.nvme0c0n1.out‚46705.2607421875áCžÏ‚(:“ µ¤ëÒ刜È gpu.0.gpu‚72.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚334.977' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.99539999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TTqîЂ(:ã µ¤ëÒò±ŽÉ disk.rbd0.out‚0 disk.drbd2165.in‚0( proc.memory.availableMB‚ 1338.0234375( disk.nvme0c0n1.out‚47377.61962890625 network.recv‚ 1.784222e+06$ disk./.usageGB‚585.7854042053223 disk.rbd1.in‚0 disk.sda.out‚ 312.4609375# disk.dm-0.out‚47327.74462890625 disk.dm-1.out‚ 312.4609375 disk.sda2.out‚0 disk.sda3.in‚ 0.47265625 cpu‚25.522824403265147 network.sent‚ 7.735875e+06! disk.nvme0n1.in‚ 9148.69921875 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚91.83335304260254 proc.cpu.threads‚17 disk.dm-0.in‚ 9148.69921875 disk.sda3.out‚ 312.4609375* proc.memory.percent‚17.884039878845215) disk./.usagePercent‚39.31254998291098 disk.drbd2165.out‚0& disk.nvme0n1.out‚47377.61962890625 disk.sda1.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2930.12109375 disk.dm-1.in‚ 0.47265625 disk.rbd1.out‚ 19.02734375 disk.sda.in‚ 0.47265625# disk.nvme0c0n1.in‚ 9148.69921875„Î’l2Ñ- _wandbruntime‚7123  _runtime‚7123Îú *ÈÒ ' lr-AdamW/pg2‚0.00044237447243830467 trainer/global_step‚4049 $ _timestamp‚1.784336962930884e+09 _runtime‚7123.515988105 _step‚330 ' lr-AdamW/pg1‚0.00044237447243830467«i˜ÍÓÇ ' lr-AdamW/pg1‚0.00044237447243830467 ' lr-AdamW/pg2‚0.00044237447243830467 trainer/global_step‚4049 $ _timestamp‚1.784336962930884e+09 _runtime‚7123.515988105 _step‚330ÊW9³2Ô- _wandbruntime‚7123  _runtime‚7123s¨™ ÕùO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1168080930571776 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4915656376320 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11525717 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1167569571598498 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1163766779342848 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3809311638016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314492824932352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4915690912768 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segq5§ûment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1163766804770816 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3809311638016 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35072052 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34558725 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319408481308672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10392534 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1167570941626788 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21918246 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35082025 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 trainer/global_step‚4049 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1164270247370752 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35082025 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69630777 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1164270247370752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3810657027584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35072052 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3810657027584 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1168079559008768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21918509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11525975 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1168079559008768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚331 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10392529 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 % _timestamp‚1.7843369629321845e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34558725 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34558731 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69640756 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1164270273544192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319408883936256 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314493193023488úŽg(ÖþO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11525975 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1168079559008768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚331 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10392529 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 % _timestamp‚1.7843369629321845e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34558725 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34558731 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69640756 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1164270273544192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319408883936256 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314493193023488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1168080930571776 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4915656376320 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11525717 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1167569571598498 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1163766779342848 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3809311638016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314492824932352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4915690912768 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1163766804770816 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3809311638016 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35072052 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34558725 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319408481308672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10392534 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1167570941626788 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21918246 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35082025 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 trainer/global_step‚4049 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1164270247370752 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35082025 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69630777 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1164270247370752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3810657027584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35072052 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3810657027584 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1168079559008768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21918509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979Ë\€†2×- _wandbruntime‚7124  _runtime‚7124ÿˆÝ '؃N REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35090517 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314570311340032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1164548248100864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10395072 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚4049 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1164548248100864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21923816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1168359856344576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1168358484774400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3803715721896 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843369643575807e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35090517 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3810236673536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3810236673536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11528739 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35080530 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69647461 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35080530 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10395077 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319487171999744 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _step‚332 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34566937 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314570679431168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69657454 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1164548274274304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4916893091840 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚7124.943011097 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3811582070272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69647461 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4916860659712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34566931 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21923568 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319487572523008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3805060322330 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11528496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34566931 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21923568 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319487572523008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3805060322330 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11528496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1164548248100864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21923816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1168359856344576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1168358484774400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3803715721896 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segÀkment.small_pool.peak‚820 % _timestamp‚1.7843369643575807e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35090517 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3810236673536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3810236673536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11528739 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35080530 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69647461 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35080530 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10395077 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319487171999744 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _step‚332 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34566937 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314570679431168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69657454 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1164548274274304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4916893091840 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚7124.943011097 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3811582070272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69647461 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4916860659712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1Ì}s¢»“Ú‚(:ˆ ĤëÒ»šÒÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚289.215' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.843 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ãôe2Û- _wandbruntime‚7125  _runtime‚7125>w¶Þ Ü… 1loss/SRqqt1/detection_loss‚0.7835952955031328 0loss/FBt1/detection_loss‚0.46306795973941917 /loss/FBt2/detection_loss‚0.5242025441179673 0loss/FRt2/assignment_loss‚0.8388749503541266 3loss/FBt2/assignment_loss‚0.0036663289656795922 /loss/FRt1/detection_loss‚0.5504096439494255 .loss/FRt2/detection_loss‚1.581346234614993 1loss/SRqqt2/detection_loss‚0.7902806533777239 epoch‚10 trainer/global_step‚4049 %loss/total_loss‚8.212082215074776 2loss/SRqqt1/assignment_loss‚0.5121574513601247 2loss/SRqqt2/assignment_loss‚0.5333897330131561 2loss/FBt1/assignment_loss‚0.008158099037222144 % _timestamp‚1.7843369644206338e+09 _runtime‚7125.005866144 _step‚333 0loss/FRt1/assignment_loss‚1.6229333210418049{µCÝŠ /loss/FBt2/detection_loss‚0.5242025441179673 0loss/FRt2/assignment_loss‚0.8388749503541266 3loss/FBt2/assignment_loss‚0.0036663289656795922 /loss/FRt1/detection_loss‚0.5504096439494255 .loss/FRt2/detection_loss‚1.581346234614993 1loss/SRqqt2/detection_loss‚0.7902806533777239 epoch‚10 trainer/global_step‚4049 %loss/total_loss‚8.212082215074776 2loss/SRqqt1/assignment_loss‚0.5121574513601247 2loss/SRqqt2/assignment_loss‚0.5333897330131561 2loss/FBt1/assignment_loss‚0.008158099037222144 % _timestamp‚1.7843369644206338e+09 _runtime‚7125.005866144 _step‚333 0loss/FRt1/assignment_loss‚1.6229333210418049 1loss/SRqqt1/detection_loss‚0.7835952955031328 0loss/FBt1/detection_loss‚0.46306795973941917Í-ÈwåÞ‚(:Ú ĤëÒ¡éÊ disk.nvme0n1.in‚ 9149.53125 disk.sda3.out‚ 313.14453125& proc.memory.availableMB‚ 1338.28125 disk.sda1.in‚0$ memory_percent‚91.83177947998047 network.sent‚ 7.740896e+06 disk.rbd0.in‚0 disk.dm-1.out‚ 313.14453125 disk.sda.out‚ 313.14453125( disk.nvme0c0n1.out‚47581.37451171875& disk.nvme0n1.out‚47581.37451171875# proc.memory.rssMB‚ 2930.12109375 network.recv‚ 1.787634e+06 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.47265625 disk.sda1.out‚0) disk./.usagePercent‚39.26383482123546# disk./.usageGB‚585.059513092041 disk.rbd0.out‚0 disk.rbd1.out‚ 19.08984375 disk.sda.in‚ 0.47265625* proc.memory.percent‚17.884039878845215 cpu‚25.522414210419033 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 9149.53125# disk.dm-0.out‚47581.37451171875 disk.sda3.in‚ 0.47265625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 9149.53125 disk.drbd2165.in‚0AUW7žß‚(:“ Ó¤ëÒ×òÉ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚291.095' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.21900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0† *èà‚(:Ý Ó¤ëÒ²»èÉ disk.dm-0.in‚ 9150.51171875& disk.nvme0n1.out‚47589.31884765625 disk.sda2.out‚0 disk.sda.out‚ 313.71875# disk.nvme0c0n1.in‚ 9150.51171875 disk.sda1.out‚0# proc.memory.rssMB‚ 2930.12109375* disk./.usagePercent‚39.266452754280614$ disk./.usageGB‚585.0985221862793 proc.cpu.threads‚17! disk.nvme0n1.in‚ 9150.51171875 disk.sda2.in‚0 disk.sda.in‚ 0.47265625( disk.nvme0c0n1.out‚47589.31884765625# disk.dm-0.out‚47589.31884765625( proc.memory.availableMB‚ 1338.0390625 network.recv‚ 1.792649e+06 disk.sda1.in‚0 disk.sda3.out‚ 313.71875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 313.71875 disk.rbd1.out‚ 19.08984375 cpu‚25.522060807326383 disk.sda3.in‚ 0.47265625 disk.rbd1.in‚0 disk.drbd2165.out‚0# memory_percent‚91.8332576751709* proc.memory.percent‚17.884039878845215 network.sent‚ 7.803177e+06 disk.dm-1.in‚ 0.47265625 disk.drbd2165.in‚0­Mò”á‚(:‰ â¤ëÒÁÑÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚141.211' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.2422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QÊ9àåâ‚(:Ú â¤ëÒùÞËÉ disk.sda2.in‚0 disk.dm-1.out‚ 314.28125 disk.sda.out‚ 314.28125" disk.dm-0.out‚47595.7412109375 proc.cpu.threads‚17* disk./.usagePercent‚39.262012303599214 disk.rbd0.out‚0' disk.nvme0c0n1.out‚47595.7412109375( proc.memory.availableMB‚ 1337.9453125* proc.memory.percent‚17.884039878845215 cpu‚25.521807344691254! disk.nvme0n1.in‚ 9150.51171875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.sent‚ 7.845482e+06 disk.sda1.out‚0# disk./.usageGB‚585.032356262207 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 9150.51171875 disk.dm-0.in‚ 9150.51171875# proc.memory.rssMB‚ 2930.12109375$ memory_percent‚91.83382987976074 network.recv‚ 1.796744e+06 disk.sda3.in‚ 0.47265625 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0 disk.rbd1.out‚ 19.23046875 disk.sda.in‚ 0.47265625% disk.nvme0n1.out‚47595.7412109375 disk.sda3.out‚ 314.28125¹kW’ã‚(:‡ ñ¤ëÒ“ˆ­Ç gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚109.884' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9768 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`½Íýéä‚(:Þ ñ¤ëÒ€úºÉ$ disk./.usageGB‚585.0312652587891 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 315.74609375 disk.sda.in‚ 0.47265625 disk.sda1.in‚0$ memory_percent‚91.83547496795654 cpu‚25.521519660385657 disk.rbd1.out‚ 19.23046875 disk.rbd0.out‚0 network.sent‚ 7.850459e+06* proc.memory.percent‚17.884039878845215) disk./.usagePercent‚39.26193908544442 disk.nvme0n1.in‚ 9233.46875& disk.nvme0n1.out‚47654.83544921875 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda3.out‚ 315.74609375 network.recv‚ 1.800067e+06 disk.sda.out‚ 315.74609375( disk.nvme0c0n1.out‚47654.83544921875) proc.memory.availableMB‚ 1337.67578125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 9233.46875# disk.dm-0.out‚47654.83544921875# proc.memory.rssMB‚ 2930.12109375 disk.nvme0c0n1.in‚ 9233.46875 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0w” œå‚(:‘ €¥ëÒû×ûÇ gpu.0.gpu‚2.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚301.872' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.37440000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï‚rææ‚(:Û €¥ëÒ®ŽÉÉ$ memory_percent‚91.83402061462402* disk./.usagePercent‚39.270539146542156$ disk./.usageGB‚585.1594123840332 disk.sda3.out‚ 318.41015625 disk.sda.out‚ 318.41015625 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 9233.46875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 19.29296875 disk.sda.in‚ 0.47265625 network.sent‚ 7.854549e+06 network.recv‚ 1.802637e+06 disk.sda2.out‚0 disk.rbd1.in‚0* proc.memory.percent‚17.884039878845215" disk.dm-0.out‚47679.5009765625% disk.nvme0n1.out‚47679.5009765625 disk.dm-0.in‚ 9233.46875 disk.dm-1.in‚ 0.47265625# proc.memory.rssMB‚ 2930.12109375 cpu‚25.521161994636284 proc.cpu.threads‚17' disk.nvme0c0n1.out‚47679.5009765625 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 318.41015625 disk.drbd2165.in‚0( proc.memory.availableMB‚ 1337.9140625 disk.nvme0c0n1.in‚ 9233.46875 disk.sda3.in‚ 0.47265625 disk.sda1.in‚0–]@v”ç‚(:‰ ¥ëÒŽÜöÈ gpu.0.gpu‚65.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R~oìè‚(:á ¥ëÒü¼·É proc.cpu.threads‚17 network.sent‚ 7.863421e+06 disk.rbd1.in‚0' disk.nvme0c0n1.out‚47736.0478515625! disk.nvme0n1.in‚ 9234.43359375 disk.rbd1.out‚ 19.29296875 network.recv‚ 1.813845e+06 disk.sda.out‚ 319.22265625 disk.sda3.out‚ 319.22265625 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 9234.43359375% disk.nvme0n1.out‚47736.0478515625 disk.sda1.in‚0) disk./.usagePercent‚39.27866124157346 disk.dm-1.in‚ 0.47265625 disk.drbd2165.out‚0 disk.sda3.in‚ 0.47265625 disk.sda2.out‚0 disk.dm-1.out‚ 319.22265625 disk.rbd0.out‚0$ memory_percent‚91.83416366577148' proc.memory.availableMB‚ 1337.890625# proc.memory.rssMB‚ 2930.12109375 cpu‚25.52084227825739 disk.sda1.out‚0$ disk./.usageGB‚585.2804374694824 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 9234.43359375" disk.dm-0.out‚47736.0478515625* proc.memory.percent‚17.884039878845215*•è 2é- _wandbruntime‚7206  _runtime‚7206…WMÉêà trainer/global_step‚4099 % _timestamp‚1.7843370457067282e+09 _runtime‚7206.291937182 _step‚334 ' lr-AdamW/pg1‚0.00044130062944156353 ' lr-AdamW/pg2‚0.00044130062944156353%*¶åÎëÈ ' lr-AdamW/pg2‚0.00044130062944156353 trainer/global_step‚4099 % _timestamp‚1.7843370457067282e+09 _runtime‚7206.291937182 _step‚334 ' lr-AdamW/pg1‚0.00044130062944156353ζ CÖ2ì- _wandbruntime‚7206  _runtime‚7206¬+™$þ'íøO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1182027283008000 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1178170310049792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚7206.293839113 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚318367145316352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3849029368758 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34969031 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34969025 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35507777 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70466829 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1178170283876352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35497804 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35507777 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22184529 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1177660893679616 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1182025911444992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚323342896440320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784337045708294e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70476808 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚318367513407488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4975383032832 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35497804 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10519679 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _step‚335 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34969025 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚323342491715584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3850373969080 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1178170283876352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34969031 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11665122 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1181509897620406 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35507777 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70466829 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1178170283876352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35497804 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35507777 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22184529 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1177660893679616 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1182025911444992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚323342896440320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784337045708294e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70476808 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚318367513407488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4975383032832 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35497804 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10519679 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _step‚335 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34969025 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚323342491715584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3850373969080 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1178170283876352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34969031 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11665122 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1181509897620406 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70493506 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843370471275394e+09 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3857898000896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1182306208780800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70483513 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1182306208780800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3856552604160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10522222 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚323421167095808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1178448310779904 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3856552604160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34977231 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35506282 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4976571998208 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1178448284606464 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11667887 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚318444999815168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1177938775457792 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _runtime‚7207.712992145 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34977237 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35516269 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1177938750029824 trainer/global_step‚4099 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70493506 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22190114 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3851297435438 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3849952835004 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35516269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚318444631724032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22189846 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70483513 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35506282 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22190114 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3851297435438 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3849952835004 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35516269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚318444631724032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22189846 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70483513 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35506282 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70493506 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843370471275394e+09 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3857898000896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1182306208780800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70483513 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1182306208780800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3856552604160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10522222 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚323421167095808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1178448310779904 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3856552604160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6ÐíXIh2ò- _wandbruntime‚7207  _runtime‚7207ÒŒ-`ó‚ /loss/FRt1/detection_loss‚0.5123797951842355 %loss/total_loss‚8.786590428372396 % _timestamp‚1.7843370472038157e+09 _runtime‚7207.788918962 /loss/FRt2/assignment_loss‚0.894133211714443 1loss/SRqqt2/assignment_loss‚0.543948768236315 1loss/SRqqt2/detection_loss‚0.7516452495733069 trainer/global_step‚4099 epoch‚10 1loss/FBt1/assignment_loss‚0.02993638628386469 /loss/FRt2/detection_loss‚1.7280864130949798 1loss/SRqqt1/detection_loss‚0.6735736333786088 /loss/FBt1/detection_loss‚0.5151328278334404 /loss/FBt2/detection_loss‚0.8898652023786018 _step‚337 2loss/SRqqt1/assignment_loss‚0.5070371147471476 3loss/FBt2/assignment_loss‚0.0030256754807509874 0loss/FRt1/assignment_loss‚1.7378261504667019b…‹ ô‡ epoch‚10 1loss/FBt1/assignment_loss‚0.02993638628386469 /loss/FRt2/detection_loss‚1.7280864130949798 1loss/SRqqt1/detection_loss‚0.6735736333786088 /loss/FBt1/detection_loss‚0.5151328278334404 /loss/FBt2/detection_loss‚0.8898652023786018 _step‚337 2loss/SRqqt1/assignment_loss‚0.5070371147471476 3loss/FBt2/assignment_loss‚0.0030256754807509874 0loss/FRt1/assignment_loss‚1.7378261504667019 /loss/FRt1/detection_loss‚0.5123797951842355 %loss/total_loss‚8.786590428372396 % _timestamp‚1.7843370472038157e+09 _runtime‚7207.788918962 /loss/FRt2/assignment_loss‚0.894133211714443 1loss/SRqqt2/assignment_loss‚0.543948768236315 1loss/SRqqt2/detection_loss‚0.7516452495733069 trainer/global_step‚4099ÑL5ÉN”õ‚(:‰ ž¥ëÒ½”Ç gpu.0.gpu‚66.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚325.017' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.0034 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡'Y)ãö‚(:Ø ž¥ëÒˆ´äÉ disk.sda1.in‚0& proc.memory.availableMB‚ 1337.65625 disk.sda.in‚ 0.47265625 disk.dm-0.in‚ 9235.265625 disk.sda1.out‚0 disk.rbd1.out‚ 19.37109375# proc.memory.rssMB‚ 2930.12109375 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 319.81640625 network.sent‚ 7.88158e+06 disk.sda.out‚ 319.81640625 disk.dm-1.in‚ 0.47265625) disk./.usagePercent‚39.27470643718451# disk./.usageGB‚585.221508026123 disk.nvme0n1.in‚ 9235.265625% disk.nvme0n1.out‚47838.7568359375 disk.sda2.in‚0 disk.sda2.out‚0# memory_percent‚91.8355941772461' disk.nvme0c0n1.out‚47838.7568359375 disk.rbd0.out‚0 disk.rbd1.in‚0 cpu‚25.520554835963157 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0" disk.dm-0.out‚47838.7568359375 disk.rbd0.in‚0 disk.dm-1.out‚ 319.81640625* proc.memory.percent‚17.884039878845215 network.recv‚ 1.817585e+06! disk.nvme0c0n1.in‚ 9235.265625:5)”÷‚(:‰ ­¥ëÒ”•ºÈ gpu.0.gpu‚85.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚332.413' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I÷£äãø‚(:Ø ­¥ëÒ£ŸŽÊ disk.sda3.in‚ 0.47265625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625 disk.sda1.in‚0# proc.memory.rssMB‚ 2930.12109375* proc.memory.percent‚17.884039878845215 cpu‚25.520234229131347* disk./.usagePercent‚39.274359802983156 disk.dm-0.in‚ 9299.07421875$ disk.nvme0n1.out‚48086.677734375& proc.memory.availableMB‚ 1337.65625 network.sent‚ 7.931896e+06 network.recv‚ 1.823012e+06$ disk./.usageGB‚585.2163429260254 disk.sda.out‚ 321.546875& disk.nvme0c0n1.out‚48086.677734375 disk.rbd0.in‚0 disk.dm-1.out‚ 321.546875 disk.rbd1.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 9299.07421875 proc.cpu.threads‚17! disk.dm-0.out‚48086.677734375! disk.nvme0n1.in‚ 9299.07421875 disk.rbd0.out‚0 disk.sda.in‚ 0.47265625 disk.sda1.out‚0# memory_percent‚91.8355941772461 disk.sda3.out‚ 321.546875 disk.rbd1.out‚ 19.37109375 disk.drbd2165.in‚0´o‘ù‚(:† ¼¥ëÒ¬êÄÈ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚338.45' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.69 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mëSÐèú‚(:Ý ¼¥ëÒÃü¹É& proc.memory.availableMB‚ 1337.84375 disk.dm-1.out‚ 322.62109375 disk.sda.in‚ 0.47265625( disk.nvme0c0n1.out‚48095.71435546875* proc.memory.percent‚17.884039878845215 cpu‚25.51992228628545 disk.drbd2165.in‚0# memory_percent‚91.8344497680664& disk.nvme0n1.out‚48095.71435546875 disk.sda2.out‚0 disk.sda.out‚ 322.62109375( disk./.usagePercent‚39.2743915479174 disk.rbd0.out‚0 disk.dm-0.in‚ 9300.0390625# proc.memory.rssMB‚ 2930.12109375 disk.nvme0n1.in‚ 9300.0390625 disk.sda3.out‚ 322.62109375 disk.rbd1.in‚0 network.recv‚ 1.83389e+06 disk.sda1.out‚0 disk.rbd1.out‚ 19.43359375" disk.nvme0c0n1.in‚ 9300.0390625# disk.dm-0.out‚48095.71435546875 proc.cpu.threads‚17 network.sent‚ 7.940422e+06$ disk./.usageGB‚585.2168159484863 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 disk.drbd2165.out‚0Û’X”û‚(:‰ Ë¥ëÒðÉÓÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚342.114' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.4228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òy{hïü‚(:ä Ë¥ëÒƒ¨ÝÉ disk.rbd0.in‚0) proc.memory.availableMB‚ 1337.83984375 network.sent‚ 7.945278e+06) disk./.usagePercent‚39.26619060256555 disk.sda2.in‚0 disk.dm-0.in‚ 9300.87109375! disk.nvme0n1.in‚ 9300.87109375 cpu‚25.519604644858962 disk.dm-1.out‚ 323.2109375 disk.sda.in‚ 0.47265625 disk.sda.out‚ 323.2109375( disk.nvme0c0n1.out‚48108.43310546875# proc.memory.rssMB‚ 2930.12109375 network.recv‚ 1.837234e+06 disk.rbd1.out‚ 19.43359375 disk.sda1.in‚0 disk.sda3.out‚ 323.2109375 disk.rbd0.out‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 9300.87109375# disk.dm-0.out‚48108.43310546875 disk.sda1.out‚0$ disk./.usageGB‚585.0946159362793 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0 disk.sda3.in‚ 0.47265625$ memory_percent‚91.83447360992432* proc.memory.percent‚17.884039878845215 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.drbd2165.in‚0& disk.nvme0n1.out‚48108.43310546875è8rnŸý‚(:” Ú¥ëÒÝ’²Ç gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚182.554' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.510799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xÒjÇèþ‚(:Ý Ú¥ëÒýÞÊ* proc.memory.percent‚17.884039878845215) disk./.usagePercent‚39.26908758382301 disk.dm-1.out‚ 323.828125 disk.sda1.out‚0# memory_percent‚91.8344497680664 cpu‚25.519248933250413 proc.cpu.threads‚17 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 9300.87109375( disk.nvme0c0n1.out‚48111.20751953125 disk.dm-0.in‚ 9300.87109375# disk.dm-0.out‚48111.20751953125 disk.drbd2165.out‚0 disk.sda3.in‚ 0.47265625 network.recv‚ 1.840354e+06 disk.sda.out‚ 323.828125# proc.memory.rssMB‚ 2930.12109375 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 0.47265625 disk.rbd1.out‚ 19.51171875 disk.sda.in‚ 0.47265625! disk.nvme0n1.in‚ 9300.87109375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk./.usageGB‚585.1377830505371& disk.nvme0n1.out‚48111.20751953125 disk.sda3.out‚ 323.828125 network.sent‚ 7.950289e+06 disk.sda1.in‚0& proc.memory.availableMB‚ 1337.84375<¼l?ÿ‚(:“ é¥ëÒÓÒØÇ gpu.0.gpu‚99.0 gpu.0.memory‚Ĭ´d_29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚283.847' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.76939999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~þb¨è€‚(:Ý é¥ëÒÚàšÉ disk.rbd1.in‚0 network.recv‚ 1.843731e+06 disk.nvme0c0n1.in‚ 9366.21875 disk.dm-0.in‚ 9366.21875 disk.rbd0.out‚0 disk.sda2.in‚0 cpu‚25.51887873415797) disk./.usagePercent‚39.26583142399501 disk.nvme0n1.in‚ 9366.21875 disk.sda3.in‚ 0.47265625& disk.nvme0n1.out‚48237.61279296875 disk.sda2.out‚0 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1337.78515625 disk.rbd1.out‚ 19.51171875 disk.sda.out‚ 325.37109375$ memory_percent‚91.83480739593506 network.sent‚ 7.955268e+06( disk.nvme0c0n1.out‚48237.61279296875 disk.dm-1.out‚ 325.37109375 proc.cpu.threads‚17$ disk./.usageGB‚585.0892639160156 disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0* proc.memory.percent‚17.884039878845215# disk.dm-0.out‚48237.61279296875 disk.sda1.out‚0 disk.sda3.out‚ 325.37109375 disk.dm-1.in‚ 0.47265625# proc.memory.rssMB‚ 2930.12109375í²Òû2- _wandbruntime‚7290  _runtime‚7290)&ÒLÇ‚Á & lr-AdamW/pg1‚0.0004402139367123641 & lr-AdamW/pg2‚0.0004402139367123641 trainer/global_step‚4149 % _timestamp‚1.7843371294936585e+09 _runtime‚7290.078906422 _step‚338åa÷¤ÌƒÆ & lr-AdamW/pg1‚0.0004402139367123641 & lr-AdamW/pg2‚0.0004402139367123641 trainer/global_step‚4149 % _timestamp‚1.7843371294936585e+09 _runtime‚7290.078906422 _step‚338Òq+y$2„- _wandbruntime‚7290  _runtime‚7290ôPf­ÿ'…ùO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35379331 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5034620332032 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1195972263881216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11802034 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11802298 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322241465700352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843371294953194e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322241833791488 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1195450223642314 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3903288888832 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1192070346555392 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35923556 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3895266481866 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10646834 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3901943499264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71312860 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35933529 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1195973635444224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22448863 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3896611082188 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _runtime‚7290.080900587 trainer/global_step‚4149 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10646829 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚327276454123520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71302881 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3901943499264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35379325 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1191554982588416 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1192070320381952 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35933529 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1192070346555392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35923556 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3903288888832 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _step‚339 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1192070320381952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35379325 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚327276049398784 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1191554957160448 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1195972263881216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1195973635444224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22449132 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5034583698432 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1195451593670604 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35379331 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11802298 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322241465700352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843371294953194e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322241833791488 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1195450223642314 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3903288888832 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1192070346555392 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35923556 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3895266481866 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10646834 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3901943499264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71312860 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35933529 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1195973635444224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22448863 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3896611082188 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _runtime‚7290.080900587 trainer/global_step‚4149 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10646829 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚327276454123520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71302881 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3901943499264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35379325 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1191554982588416 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1192070320381952 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35933529 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1192070346555392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35923556 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3903288888832 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _step‚339 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1192070320381952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35379325 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚327276049398784 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1191554957160448 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1195972263881216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1195973635444224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22449132 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5034583698432 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1195451593670604 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35379331 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1195972263881216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11802034 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175ÓÕW€|2‡- _wandbruntime‚7291  _runtime‚7291r¨Ì'ˆ‚N PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚327354712228864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843371309495704e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71329558 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11805032 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1192348347285504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11804773 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10649372 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10649377 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1195729028886736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚ 7291.53519901 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322319320199168 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚327355116946432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71319565 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3904213931520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35942021 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5035796747264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22454145_þ‚8‰‡N RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5035796747264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.al^(³áÕ%l.peak‚1995 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22454145 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚327354712228864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843371309495704e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71329558 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11805032 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1192348347285504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11804773 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10649372 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10649377 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1195729028886736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚ 7291.53519901 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322319320199168 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚327355116946432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71319565 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3904213931520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35942021 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995ÔW£Ì2Š- _wandbruntime‚7291  _runtime‚7291‘•’‹þ 0loss/FRt2/assignment_loss‚0.8283297854458566 2loss/SRqqt1/assignment_loss‚0.5495258379302362 1loss/FBt2/assignment_loss‚0.04358489953497293 /loss/FRt2/detection_loss‚1.7225546807361145 % _timestamp‚1.7843371309935653e+09 2loss/SRqqt2/assignment_loss‚0.5588638296200139 /loss/FRt1/detection_loss‚0.5296773636393731 0loss/SRqqt1/detection_loss‚0.783625035928627 1loss/SRqqt2/detection_loss‚0.6449185774363518 /loss/FBt2/detection_loss‚0.7329850703985854 _runtime‚7291.578742959 /loss/FBt1/detection_loss‚0.5486294812218723 %loss/total_loss‚8.653543072064364 trainer/global_step‚4149 2loss/FBt1/assignment_loss‚0.013649568478360432 epoch‚10 _step‚341 ,loss/FRt1/assignment_loss‚1.697198941694ر r Œƒ 2loss/SRqqt1/assignment_loss‚0.5495258379302362 1loss/FBt2/assignment_loss‚0.04358489953497293 /loss/FRt2/detection_loss‚1.7225546807361145 % _timestamp‚1.7843371309935653e+09 2loss/SRqqt2/assignment_loss‚0.5588638296200139 /loss/FRt1/detection_loss‚0.5296773636393731 0loss/SRqqt1/detection_loss‚0.783625035928627 1loss/SRqqt2/detection_loss‚0.6449185774363518 /loss/FBt2/detection_loss‚0.7329850703985854 _runtime‚7291.578742959 /loss/FBt1/detection_loss‚0.5486294812218723 %loss/total_loss‚8.653543072064364 trainer/global_step‚4149 2loss/FBt1/assignment_loss‚0.013649568478360432 epoch‚10 _step‚341 ,loss/FRt1/assignment_loss‚1.697198941694 0loss/FRt2/assignment_loss‚0.8283297854458566ÕÊ=BŸ‚(:” ø¥ëÒÐéáÇ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›±9G玂(:Ü ø¥ëÒœ÷©É disk.rbd1.out‚ 19.57421875 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 9367.05078125 disk.dm-0.out‚48246.21484375 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 326.63671875# proc.memory.rssMB‚ 2930.12109375* proc.memory.percent‚17.884039878845215 disk.rbd0.out‚0 disk.dm-1.out‚ 326.63671875 disk.drbd2165.in‚0% disk.nvme0c0n1.out‚48246.21484375 disk.sda1.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 1337.390625 network.recv‚ 1.847308e+06 disk.sda2.in‚0# disk.nvme0n1.out‚48246.21484375$ memory_percent‚91.83721542358398 proc.cpu.threads‚17* disk./.usagePercent‚39.265867265049806 disk.dm-0.in‚ 9367.05078125! disk.nvme0n1.in‚ 9367.05078125 network.sent‚ 7.960392e+06 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 326.63671875 disk.sda.in‚ 0.47265625 cpu‚25.51860048330599$ disk./.usageGB‚585.0897979736328 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625ßJè¼”‚(:‰ ‡¦ëÒáÊ›Ç gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚239.187' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.8374 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢X¢nð‚(:å ‡¦ëÒ´‚òÊ disk.sda2.in‚0 network.recv‚ 1.852383e+06 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625* proc.memory.percent‚17.884039878845215! disk.nvme0n1.in‚ 9367.05078125& disk.nvme0n1.out‚48254.96142578125 disk.rbd1.in‚0# proc.memory.rssMB‚ 2930.12109375 cpu‚25.51831301933065 proc.cpu.threads‚17 disk.rbd1.out‚ 19.57421875) disk./.usagePercent‚39.30083763821949 disk.sda.out‚ 327.23046875# disk.nvme0c0n1.in‚ 9367.05078125 disk.sda2.out‚0$ memory_percent‚91.83640480041504$ disk./.usageGB‚585.6108818054199 disk.sda.in‚ 0.47265625# disk.dm-0.out‚48254.96142578125 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 9367.05078125 disk.sda3.out‚ 327.23046875 disk.dm-1.out‚ 327.23046875( proc.memory.availableMB‚ 1337.5234375 network.sent‚ 8.023414e+06 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚48254.96142578125 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0+¯öŸ‘‚(:” –¦ëÒ§¼ÍÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚273.925' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.785000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dƒ°Äå’‚(:Ú –¦ëÒ´¢ØÉ! disk.nvme0c0n1.in‚ 9368.015625( proc.memory.availableMB‚ 1337.2734375 disk.rbd0.in‚0 disk.sda2.in‚0 network.recv‚ 1.85578e+06* disk./.usagePercent‚39.319376423811505 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd2165.out‚0 disk.sda.in‚ 0.47265625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 9368.015625% disk.nvme0n1.out‚48290.1396484375# proc.memory.rssMB‚ 2930.12109375 disk.drbd2165.in‚0 disk.rbd1.out‚ 19.63671875 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625 disk.sda.out‚ 327.6796875' disk.nvme0c0n1.out‚48290.1396484375" disk.dm-0.out‚48290.1396484375 network.sent‚ 8.028327e+06$ disk./.usageGB‚585.8871231079102 disk.dm-1.out‚ 327.6796875 disk.dm-0.in‚ 9368.015625$ memory_percent‚91.83793067932129 disk.sda3.out‚ 327.6796875* proc.memory.percent‚17.884039878845215 cpu‚25.518012540992718y0<€Ÿ“‚(:” ¥¦ëÒÿÿöÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚284.382' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.876400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒÓqð”‚(:å ¥¦ëÒ¸„ÏÉ# disk.dm-0.out‚48301.09912109375) proc.memory.availableMB‚ 1337.10546875 network.sent‚ 8.033413e+06 disk.sda3.out‚ 328.3828125 disk.dm-1.in‚ 0.47265625( disk.nvme0c0n1.out‚48301.09912109375* proc.memory.percent‚17.884039878845215 network.recv‚ 1.859352e+06$ disk./.usageGB‚585.8588905334473& disk.nvme0n1.out‚48301.09912109375 disk.sda3.in‚ 0.47265625 disk.dm-1.out‚ 328.3828125# disk.nvme0c0n1.in‚ 9432.42578125 disk.dm-0.in‚ 9432.42578125 disk.sda1.out‚0 disk.rbd1.out‚ 19.63671875 disk.sda.in‚ 0.47265625* disk./.usagePercent‚39.317481712050586! disk.nvme0n1.in‚ 9432.42578125 disk.rbd0.in‚0 disk.sda.out‚ 328.3828125$ memory_percent‚91.83895587921143 cpu‚25.517705251222907 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0# proc.memory.rssMB‚ 2930.12109375îâçÁž•‚(:“ ´¦ëÒÊ¥ñÈ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚294.438' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.88759999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôqê–‚(:ß ´¦ëÒè¥àÉ$ disk./.usageGB‚584.9756393432617 disk.sda2.out‚0 disk.rbd1.out‚ 19.71484375 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda.in‚ 0.47265625 disk.sda3.in‚ 0.47265625 disk.sda.out‚ 329.48046875 disk.sda3.out‚ 329.48046875) proc.memory.availableMB‚ 1336.85546875 cpu‚25.51743154949705$ memory_percent‚91.84048175811768) disk./.usagePercent‚39.25820598358013 disk.dm-0.in‚ 9432.42578125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625# disk.nvme0c0n1.in‚ 9432.42578125$ disk.nvme0n1.out‚49234.673828125 disk.sda1.out‚0# proc.memory.rssMB‚ 2930.12109375 disk.dm-1.out‚ 329.48046875 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 9432.42578125 proc.cpu.threads‚17 network.sent‚ 8.038435e+06 network.recv‚ 1.86247e+06 disk.rbd1.in‚0& disk.nvme0c0n1.out‚49234.673828125! disk.dm-0.out‚49234.673828125 disk.sda1.in‚0* proc.memory.percent‚17.884039878845215Á³Y›2—- _wandbruntime‚7373  _runtime‚7373õ×Úɘà % _timestamp‚1.7843372130869148e+09 _runtime‚7373.672039381 _step‚342 ' lr-AdamW/pg1‚0.00043911446404694957 ' lr-AdamW/pg2‚0.00043911446404694957 trainer/global_step‚4199j£NÎ™È % _timestamp‚1.7843372130869148e+09 _runtime‚7373.672039381 _step‚342 ' lr-AdamW/pg1‚0.00043911446404694957 ' lr-AdamW/pg2‚0.00043911446404694957 trainer/global_step‚4199ÖCGå2š- _wandbruntime‚7373  _runtime‚7373›ê$ÿ'›ùO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36359281 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3949604819456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10773984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11940452 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11940185 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326115786084352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36359281 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35789631 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331209955706880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36349308 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1205970383060992 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1205970356887552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22714436 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5093801531392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1209391919692512 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72138933 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326116154175488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3949604819456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35789625 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1209918616317440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3941503594974 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1209919987880448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚7373.673628381 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35789625 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3948259429888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36349308 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5093764897792 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35789631 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1205449046069248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1205970383060992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10773979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568ƒM•ZÁ œþO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843372130882792e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331209550982144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1205449071497216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1209919987880448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3942848195296 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚4199 _step‚343 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1205970356887552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72148912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1209918616317440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3941503594974 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1209919987880448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚7373.673628381 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35789625 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZHDeviceStatsMonitor.om|½MCn_train_batch_start/allocated_bytes.small_pool.freed‚ 3948259429888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36349308 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5093764897792 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35789631 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1205449046069248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1205970383060992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10773979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36359281 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3949604819456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10773984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11940452 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11940185 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326115786084352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36359281 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35789631 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331209955706880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36349308 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1205970383060992 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1205970356887552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22714436 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5093801531392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1209391919692512 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72138933 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326116154175488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3949604819456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35789625 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3949184465408 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36357786 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11942769 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3943771661654 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72165610 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5094987120640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1205726953275392 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1206248357617664 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10776522 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3950529862144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3949184465408 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35797837 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1206248357617664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚331288222986240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35797831 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 % _timestamp‚1.7843372145107424e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36367773 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22719554 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1210198913653248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36357786 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1209669354908644 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7375.096489293 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1210198913653248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5094950494208 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72165610 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22719291 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326193640583168 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72155617 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1206248383791104þ€»ç'ŸˆN F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1206248357617664 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10776522 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3950529862144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3949184465408 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35797837 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1206248357617664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚331288222986240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35797831 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 % _timestamp‚1.7843372145107424e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36367773 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22719554 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1210198913653248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36357786 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1209669354908644 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7375.096489293 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1210198913653248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5094950494208 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72165610 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22719291 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326193640583168 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72155617 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1206248383791104 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10776527 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1205726927847424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1210197542083072 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35797837 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36367773 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72155617 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326193272492032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3942427061220 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 trainer/global_step‚4199 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3949184465408 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36357786 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11942769 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3943771661654 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72165610 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5094987120640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1205726953275392 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0Ø(ž2 - _wandbruntime‚7375  _runtime‚7375E2íý ¡ƒ 0loss/FRt1/assignment_loss‚1.6285429762469397 2loss/FBt2/assignment_loss‚0.009142694316946023 /loss/FRt1/detection_loss‚0.5009967048109064 /loss/FRt2/detection_loss‚1.6701963871931857 _step‚345 2loss/SRqqt2/assignment_loss‚0.5628568054698874 1loss/FBt1/assignment_loss‚0.02030346904842068 1loss/SRqqt1/detection_loss‚0.7227319698000415 1loss/SRqqt2/detection_loss‚0.6734403018049444 0loss/FBt1/detection_loss‚0.47569588739641533 epoch‚10 trainer/global_step‚4199 % _timestamp‚1.7843372145787845e+09 2loss/SRqqt1/assignment_loss‚0.4582723136740142 $loss/total_loss‚8.44605660904749 _runtime‚7375.164314488 0loss/FRt2/assignment_loss‚0.8898406358931562 /loss/FBt2/detection_loss‚0.8340364633926324y?Œ˜¢ˆ 0loss/FBt1/detection_loss‚0.47569588739641533 epoch‚10 trainer/global_step‚4199 % _timestamp‚1.7843372145787845e+09 2loss/SRqqt1/assignment_loss‚0.4582723136740142 $loss/total_loss‚8.44605660904749 _runtime‚7375.164314488 0loss/FRt2/assignment_loss‚0.8898406358931562 /loss/FBt2/detection_loss‚0.8340364633926324 0loss/FRt1/assignment_loss‚1.6285429762469397 2loss/FBt2/assignment_loss‚0.009142694316946023 /loss/FRt1/detection_loss‚0.5009967048109064 /loss/FRt2/detection_loss‚1.6701963871931857 _step‚345 2loss/SRqqt2/assignment_loss‚0.5628568054698874 1loss/FBt1/assignment_loss‚0.02030346904842068 1loss/SRqqt1/detection_loss‚0.7227319698000415 1loss/SRqqt2/detection_loss‚0.6734403018049444Ù[;«ž£‚(:“ æëÒ†„È gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚283.847' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.76939999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C û¤ñ¤‚(:æ æëÒóåËÉ$ memory_percent‚91.83886051177979 disk.drbd2165.in‚0# disk.dm-0.out‚49246.26513671875! disk.nvme0n1.in‚ 9433.41796875 disk.dm-1.out‚ 330.87890625 disk.sda3.out‚ 330.87890625 cpu‚25.517127292540387 disk.drbd2165.out‚0 network.recv‚ 1.865908e+06 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625# proc.memory.rssMB‚ 2930.12109375 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.47265625) proc.memory.availableMB‚ 1337.12109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 19.71484375( disk.nvme0c0n1.out‚49246.26513671875 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 8.043315e+06( disk./.usagePercent‚39.2564008744562$ disk./.usageGB‚584.9487419128418 disk.sda.out‚ 330.87890625 disk.dm-0.in‚ 9433.41796875& disk.nvme0n1.out‚49246.26513671875 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 9433.41796875* proc.memory.percent‚17.884039878845215™œ˜Òž¥‚(:“ Ò¦ëÒñŒ¶È gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚275.888' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.17759999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òa$Añ¦‚(:æ Ò¦ëÒÄóÁÉ network.sent‚ 8.106567e+06$ disk./.usageGB‚584.9489555358887 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0 disk.sda.out‚ 331.83203125* proc.memory.percent‚17.884039878845215 disk.rbd0.in‚0 disk.dm-1.out‚ 331.83203125 disk.drbd2165.out‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 9439.16796875 disk.sda3.in‚ 0.47265625 disk.sda1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 9439.16796875) proc.memory.availableMB‚ 1337.12109375# proc.memory.rssMB‚ 2930.12109375 proc.cpu.threads‚17 disk.sda.in‚ 0.47265625( disk.nvme0c0n1.out‚49250.93603515625$ memory_percent‚91.83886051177979 disk.sda1.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚49250.93603515625 network.recv‚ 1.87037e+06) disk./.usagePercent‚39.25641521087812& disk.nvme0n1.out‚49250.93603515625 disk.sda3.out‚ 331.83203125 disk.dm-0.in‚ 9439.16796875 cpu‚25.516823706807543 disk.rbd1.out‚ 19.77734375 disk.drbd2165.in‚0†|g”§‚(:‰ á¦ëÒ·ìíÈ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚262.877' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â{–Ñᨂ(:Ö á¦ëÒ³½ÈÉ disk.drbd2165.out‚0 disk.dm-1.in‚ 0.47265625$ memory_percent‚91.83919429779053* proc.memory.percent‚17.884039878845215 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 19.77734375 disk.dm-0.in‚ 9506.6171875# disk.dm-0.out‚49278.20068359375 disk.sda3.in‚ 0.47265625 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 9506.6171875( disk.nvme0c0n1.out‚49278.20068359375 disk.nvme0n1.in‚ 9506.6171875 disk.rbd0.out‚0 disk.sda2.in‚0 network.recv‚ 1.873928e+06 disk.dm-1.out‚332.875 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1337.06640625* disk./.usagePercent‚39.257238531108236$ disk./.usageGB‚584.9612236022949 disk.sda1.in‚0 disk.sda.in‚ 0.47265625& disk.nvme0n1.out‚49278.20068359375 disk.sda3.out‚332.875 disk.sda2.out‚0 disk.sda.out‚332.875# proc.memory.rssMB‚ 2930.12109375 cpu‚25.516520408991678 network.sent‚ 8.111659e+06 C76©‚(:“ ð¦ëÒÒß™Ç gpu.0.gpu‚99.0 gpu.0G ·‡h.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚278.811' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.76219999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â@&ં(:Õ ð¦ëÒúøþÊ disk.rbd1.out‚ 19.85546875 network.sent‚ 8.11658e+06 network.recv‚ 1.877196e+06 disk.dm-1.in‚ 0.47265625 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚49287.97509765625# proc.memory.rssMB‚ 2930.12109375$ disk./.usageGB‚584.9616355895996# disk.dm-0.out‚49287.97509765625 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda.in‚ 0.47265625# disk.nvme0c0n1.in‚ 9507.58203125 cpu‚25.516242193100787 disk.sda2.out‚0 disk.dm-1.out‚ 333.046875 disk.rbd1.in‚0 disk.dm-0.in‚ 9507.58203125 disk.sda2.in‚0! disk.nvme0n1.in‚ 9507.58203125 disk.sda3.in‚ 0.47265625 disk.sda.out‚ 333.046875 disk.drbd2165.in‚0) disk./.usagePercent‚39.25726617992193" memory_percent‚91.839599609375 proc.memory.availableMB‚1337* proc.memory.percent‚17.884039878845215& disk.nvme0n1.out‚49287.97509765625 disk.sda3.out‚ 333.046875ø™¿0Ÿ«‚(:” ÿ¦ëÒî—ÞÇ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚255.343' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.068599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žd2%(:ã ÿ¦ëÒïû³É disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0 disk.sda.out‚ 333.41796875 disk.dm-0.in‚ 9507.58203125 disk.sda1.in‚0 network.sent‚ 8.123592e+06 network.recv‚ 1.880693e+06 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 333.41796875 disk.sda2.in‚0' disk.nvme0c0n1.out‚49297.2099609375! disk.nvme0n1.in‚ 9507.58203125 cpu‚25.515952827030066 disk.dm-1.out‚ 333.41796875" disk.dm-0.out‚49297.2099609375 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0$ disk./.usageGB‚584.9616622924805 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 9507.58203125% disk.nvme0n1.out‚49297.2099609375# memory_percent‚91.8386697769165# proc.memory.rssMB‚ 2930.12109375 proc.cpu.threads‚17) disk./.usagePercent‚39.25726797197467 disk.sda3.in‚ 0.47265625 disk.rbd1.out‚ 19.85546875 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1337.15234375* proc.memory.percent‚17.884039878845215u m0”­‚(:‰ Ž§ëÒÓ¬ÑÇ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚293.854' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.7708 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡sÔ鮂(:Þ Ž§ëÒ¡‹èÉ" disk.dm-0.out‚50225.8095703125# proc.memory.rssMB‚ 2930.12109375 disk.rbd1.in‚0 network.recv‚ 1.883786e+06 disk.sda3.out‚ 335.9765625 disk.sda2.in‚0 disk.dm-0.in‚ 9535.6015625 disk.sda1.out‚0 disk.dm-1.in‚ 0.47265625 disk.sda.out‚ 335.9765625 disk.nvme0n1.in‚ 9535.6015625 disk.sda1.in‚0* proc.memory.percent‚17.884039878845215 disk.sda2.out‚0) disk./.usagePercent‚39.41498448553381 disk.sda3.in‚ 0.47265625$ memory_percent‚91.84019565582275 disk.sda.in‚ 0.47265625" disk.nvme0c0n1.in‚ 9535.6015625) proc.memory.availableMB‚ 1336.90234375 cpu‚25.515650558490766 proc.cpu.threads‚17 disk.rbd0.in‚0% disk.nvme0n1.out‚50225.8095703125 disk.rbd0.out‚0 disk.rbd1.out‚ 19.91796875 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚50225.8095703125 network.sent‚ 8.128622e+06$ disk./.usageGB‚587.3117523193359 disk.dm-1.out‚ 335.9765625 disk.drbd2165.out‚0cØë·2¯- _wandbruntime‚7457  _runtime‚7457ðq ‰É°Ã ' lr-AdamW/pg1‚0.00043800228206239476 ' lr-AdamW/pg2‚0.00043800228206239476 trainer/global_step‚4249 % _timestamp‚1.7843372966218972e+09 _runtime‚7457.207018971 _step‚346çç™Î±È _step‚346 ' lr-AdamW/pg1‚0.00043800228206239476 ' lr-AdamW/pg2‚0.00043800228206239476 trainer/global_step‚4249 % _timestamp‚1.7843372966218972e+09 _runtime‚7457.207018971ÚœðQ2²- _wandbruntime‚7457  _runtime‚7457y%5ý'³÷O aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1219870419566592 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1223864968753664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3994575360512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚335143489148416 _step‚347 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1219870393393152 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3995920750080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36775060 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3994575360512 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1219870393393152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36785033 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚329990106468352 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36785033 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚335143084423680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1223332245714420 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72984964 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36199925 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1223864968753664 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22978946 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22978664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12077535 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1223330875686130 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5153014588928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ú =ê(´üO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3995920750080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1219870393393152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36785033 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚329990106468352 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36785033 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚335143084423680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1223332245714420 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72984964 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36199925 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1223864968753664 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22978946 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22978664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12077535 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1223330875686130 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5153014588928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1219870419566592 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1223864968753664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3994575360512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚335143489148416 _step‚347 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1219870393393152 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3995920750080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36775060 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3994575360512 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22984224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10903677 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚335222152648192 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3996845792768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36208137 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3995500396032 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1224145266089472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10903672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1220148420296704 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5154191681024 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330067960967168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72991669 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3988664174328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1220148394123264 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚335221747930624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1224143894519296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12080283 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3995500396032 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73001662 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12080547 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1223611050958954 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚7458.638781406 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72991669 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640¶š’Ÿé  F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 % _timestamp‚1.7843372980532484e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3996845792768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36783538 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22983955 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1224145266089472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1223609680930552 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3990008774762 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1220148394123264 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5154155054592 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1220148420296704 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176TYH'·ˆN TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1224145266089472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1223609680930552 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3990008774762 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1220148394123264 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5154155054592 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1220148420296704 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22984224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10903677 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚335222152648192 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3996845792768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36208137 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3995500396032 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1224145266089472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10903672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1220148420296704 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5154191681024 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330067960967168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72991669 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3988664174328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1220148394123264 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚335221747930624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1224143894519296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12080283 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3995500396032 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73001662 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12080547 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1223611050958954 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚7458.638781406 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72991669 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 % _timestamp‚1.7843372980532484e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3996845792768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36783538 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22983955 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119ÜU¨Ù2¸- _wandbruntime‚7458  _runtime‚7458"K¦¹€ 0loss/FRt1/assignment_loss‚1.6846664138457301 0loss/FRt2/assignment_loss‚0.8404664872825303 2loss/FBt1/assignment_loss‚0.017733111579546313 /loss/FBt1/detection_loss‚0.5614153803898724 epoch‚10 % _timestamp‚1.7843372981118913e+09 _runtime‚7458.697077514 2loss/SRqqt2/assignment_loss‚0.5622308735565955 0loss/FBt2/assignment_loss‚0.0487153271400386 1loss/SRqqt1/detection_loss‚0.6644989995197613 0loss/SRqqt2/detection_loss‚0.655481057359799 %loss/total_loss‚9.095978278962237 2loss/SRqqt1/assignment_loss‚0.6179824041216019 .loss/FRt2/detection_loss‚1.783453810048287 trainer/global_step‚4249 _step‚349 /loss/FRt1/detection_loss‚0.5798330557645576 /loss/FBt2/detection_loss‚1.0795013583539164@2 º… trainer/global_step‚4249 _step‚349 /loss/FRt1/detection_loss‚0.5798330557645576 /loss/FBt2/detection_loss‚1.0795013583539164 0loss/FRt1/assignment_loss‚1.6846664138457301 0loss/FRt2/assignment_loss‚0.8404664872825303 2loss/FBt1/assignment_loss‚0.017733111579546313 /loss/FBt1/detection_loss‚0.5614153803898724 epoch‚10 % _timestamp‚1.7843372981118913e+09 _runtime‚7458.697077514 2loss/SRqqt2/assignment_loss‚0.5622308735565955 0loss/FBt2/assignment_loss‚0.0487153271400386 1loss/SRqqt1/detection_loss‚0.6644989995197613 0loss/SRqqt2/detection_loss‚0.655481057359799 %loss/total_loss‚9.095978278962237 2loss/SRqqt1/assignment_loss‚0.6179824041216019 .loss/FRt2/detection_loss‚1.783453810048287ÝA®k€ž»‚(:“ §ëÒ÷âÈ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚261.525' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.30499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï,w⼂(:× §ëÒã¿ÉÉ disk.rbd1.out‚ 19.91796875 disk.nvme0n1.in‚9638.375 disk.rbd0.in‚0) proc.memory.availableMB‚ 1336.91015625 disk.sda.in‚ 0.47265625# disk.dm-0.out‚52960.08837890625 disk.sda3.out‚ 337.56640625 cpu‚25.51535411992775 disk.sda.out‚ 337.56640625 disk.drbd2165.out‚0$ memory_percent‚91.84014797210693 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0& disk.nvme0n1.out‚52960.08837890625 disk.sda1.out‚0* proc.memory.percent‚17.884039878845215 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 337.56640625( disk.nvme0c0n1.out‚52960.08837890625 disk.rbd0.out‚0 network.sent‚ 8.131544e+06$ disk./.usageGB‚589.7682342529297 disk.dm-0.in‚9638.375 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625# proc.memory.rssMB‚ 2930.12109375 proc.cpu.threads‚17 network.recv‚ 1.887114e+06 disk.sda2.in‚0 disk.nvme0c0n1.in‚9638.375) disk./.usagePercent‚39.57984104922292»ò¡m”½‚(:‰ ¬§ëÒí®žÇ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚278.227' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒHŽK⾂(:× ¬§ëÒ¶óÚÉ* proc.memory.percent‚17.884039878845215$ memory_percent‚91.84012413024902 network.recv‚ 1.891461e+06 disk.sda3.out‚ 338.14453125 disk.rbd1.in‚0 disk.nvme0c0n1.in‚9638.375 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 disk.sda.out‚ 338.14453125 disk.dm-0.in‚9638.375) disk./.usagePercent‚39.74397055151478 disk.nvme0n1.in‚9638.375& disk.nvme0n1.out‚54685.95849609375 disk.dm-1.out‚ 338.14453125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚54685.95849609375# disk.dm-0.out‚54684.70849609375# proc.memory.rssMB‚ 2930.12109375 cpu‚25.515056212544668 network.sent‚ 8.194435e+06 disk.rbd0.out‚0 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 20.05859375 disk.dm-1.in‚ 0.47265625( proc.memory.availableMB‚ 1336.9140625$ disk./.usageGB‚592.2138824462891‡r Gž¿‚(:“ »§ëÒóÁÖÈ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚286.799' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.35979999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨ @ëÀ‚(:à »§ëÒ„¤ŒÉ& disk.nvme0c0n1.out‚57808.341796875$ disk.nvme0n1.out‚57808.341796875 disk.sda3.out‚ 338.46484375 disk.sda2.out‚0# proc.memory.rssMB‚ 2930.12109375 cpu‚25.51476474243884 network.sent‚ 8.199343e+06 disk.dm-1.out‚ 338.46484375$ memory_percent‚91.84165000915527( proc.memory.availableMB‚ 1336.6640625 proc.cpu.threads‚17 disk.sda.in‚ 0.47265625! disk.dm-0.out‚57808.341796875 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0* proc.memory.percent‚17.884039878845215* disk./.usagePercent‚39.906750894101194 disk.rbd1.out‚ 20.05859375 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 9639.21484375 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2165.in‚0 network.recv‚ 1.895089e+06 disk.drbd2165.out‚0 disk.dm-0.in‚ 9639.21484375! disk.nvme0n1.in‚ 9639.21484375 disk.dm-1.in‚ 0.47265625$ disk./.usageGB‚594.6394271850586 disk.sda.out‚ 338.46484375îÛñLžÁ‚(:“ ʧëÒ›«”Ç gpu.0.gpu‚88.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚285.199' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.03980000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯p`å‚(:Ú ʧëÒöÉ$ disk./.usageGB‚597.2372016906738 disk.sda1.out‚0 disk.drbd2165.out‚0$ memory_percent‚91.84165000915527 cpu‚25.514470702936567 network.recv‚ 1.898241e+06) disk./.usagePercent‚40.08108972085102* proc.memory.percent‚17.884039878845215 proc.cpu.threads‚17( disk.nvme0c0n1.out‚60524.36279296875 disk.dm-0.in‚ 9640.1796875 disk.nvme0n1.in‚ 9640.1796875 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 disk.sda.in‚ 0.47265625# proc.memory.rssMB‚ 2930.12109375& disk.nvme0n1.out‚60524.36279296875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 338.96875 network.sent‚ 8.204516e+06# disk.dm-0.out‚60524.36279296875 disk.sda3.out‚ 338.96875 disk.sda.out‚ 338.96875 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 9640.1796875 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 20.12109375( proc.memory.availableMB‚ 1336.6640625ð70ŸÃ‚(:” Ù§ëÒ÷”ŸÇ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚263.424' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.684799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\šŠ”îÄ‚(:ã Ù§ëÒÊðÁÉ disk.dm-1.out‚ 339.33984375" disk.nvme0c0n1.in‚ 9717.1953125 disk.nvme0n1.in‚ 9717.1953125) proc.memory.availableMB‚ 1336.90234375 disk.drbd2165.out‚0$ memory_percent‚91.84019565582275 cpu‚25.51420680346983( disk.nvme0c0n1.out‚62661.44677734375# disk.dm-0.out‚62661.44677734375 disk.sda3.out‚ 339.33984375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 339.33984375 disk.drbd2165.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.dm-1.in‚ 0.47265625# proc.memory.rssMB‚ 2930.12109375 disk.sda1.out‚0 network.sent‚ 8.211443e+06 network.recv‚ 1.901584e+06) disk./.usagePercent‚40.25008208625578 disk.dm-0.in‚ 9717.1953125 disk.sda2.out‚0 disk.sda.in‚ 0.47265625 proc.cpu.threads‚17$ disk./.usageGB‚599.7553100585938 disk.sda2.in‚0 disk.rbd1.out‚ 20.12109375& disk.nvme0n1.out‚62661.44677734375* proc.memory.percent‚17.884039878845215^¬ÄË2Å- _wandbruntime‚7541  _runtime‚7541Ø¿LùÇÆÁ trainer/global_step‚4299 % _timestamp‚1.7843373804369519e+09 _runtime‚7541.022209743 _step‚350 & lr-AdamW/pg1‚0.0004368774621920703 & lr-AdamW/pg2‚0.0004368774621920703¸--ÅÌÇÆ & lr-AdamW/pg1‚0.0004368774621920703 & lr-AdamW/pg2‚0.0004368774621920703 trainer/global_step‚4299 % _timestamp‚1.7843373804369519e+09 _runtime‚7541.022209743 _step‚350ÞÜXˆµ2È- _wandbruntime‚7541  _runtime‚7541fU+ºþ'ÉøO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚339077044948480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4040891291136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1233770429898752 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5212250004992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5212213371392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1237272571736328 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1237271201708038 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4035322421512 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4042236680704 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1237811321189888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4042236680704 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37200812 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37210785 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4033977821190 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23244275 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚7541.025982273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12215715 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1233237249314816 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1233770456072192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1237811321189888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36610225 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11028279 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36610231 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23243994 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1233237223886848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37200812 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1233770456072192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1237812692752896 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11028284 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚351 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73821016 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1237812692752896 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36610225 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37210785 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4040891291136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚333864794943488 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36610231SÓ™2¯ÊýO _runtime‚7541.025982273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12215715 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1233237249314816 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1233770456072192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1ó)T%237811321189888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36610225 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11028279 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36610231 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23243994 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1233237223886848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37200812 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1233770456072192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1237812692752896 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11028284 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚351 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73821016 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1237812692752896 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36610225 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37210785 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4040891291136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚333864794943488 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36610231 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚339077044948480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4040891291136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1233770429898752 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5212250004992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5212213371392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1237272571736328 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1237271201708038 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4035322421512 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4042236680704 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1237811321189888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4042236680704 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37200812 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37210785 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4033977821190 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23244275 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0ßØ3±É2Ë- _wandbruntime‚7542  _runtime‚7542Ú6ó\ '̃N XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1234048430628864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23249288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37219277 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12218466 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37209290 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36618431 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4043161723392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5213400163840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4041816326656 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1237550006952460 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4299 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1234048456802304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339155718141440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11030822 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1233515131092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333941913260032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4041816326656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚339155313423872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73827721 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73827721 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1238090246955520 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12218734 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4043161723392 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1238091618525696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4034901287436 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73837714 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333942281351168 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36618437 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4299 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1234048456802304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339155718141440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11030822 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1233515131092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333941913260032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4041816326656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚339155313423872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73827721 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73827721 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1238090246955520 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12218734 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4043161723392 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1238091618525696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4034901287436 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73837714 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333942281351168 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36618437 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23249288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37219277 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12218466 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37209290 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36618431 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4043161723392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5213400163840àDª2Î- _wandbruntime‚7542  _runtime‚7542/&œYÏ‚ 0loss/FRt1/assignment_loss‚1.5887410498995138 /loss/FRt1/detection_loss‚0.6053888835243508 1loss/SRqqt1/detection_loss‚0.7753322768733606 1loss/SRqqt2/detection_loss‚0.7538641800197798 /loss/FBt2/detection_loss‚0.5447289461976496 1loss/FBt1/assignment_loss‚0.01094139718853606 2loss/FBt2/assignment_loss‚0.008303072530791093 _runtime‚7542.509212622 _step‚353 /loss/FRt2/detection_loss‚1.6327492923616138 epoch‚10 trainer/global_step‚4299 % _timestamp‚1.7843373819241405e+09 0loss/FRt2/assignment_loss‚0.8231929298405234 2loss/SRqqt1/assignment_loss‚0.4808249075821579 2loss/SRqqt2/assignment_loss‚0.5215307812003281 .loss/FBt1/detection_loss‚0.417593520288103 %loss/total_loss‚8.163191237506709EŽ… Ð‡ /loss/FRt2/detection_loss‚1.6327492923616138 epoch‚10 trainer/global_step‚4299 % _timestamp‚1.7843373819241405e+09 0loss/FRt2/assignment_loss‚0.8231929298405234 2loss/SRqqt1/assignment_loss‚0.4808249075821579 2loss/SRqqt2/assignment_loss‚0.5215307812003281 .loss/FBt1/detection_loss‚0.417593520288103 %loss/total_loss‚8.163191237506709 0loss/FRt1/assignment_loss‚1.5887410498995138 /loss/FRt1/detection_loss‚0.6053888835243508 1loss/SRqqt1/detection_loss‚0.7753322768733606 1loss/SRqqt2/detection_loss‚0.7538641800197798 /loss/FBt2/detection_loss‚0.5447289461976496 1loss/FBt1/assignment_loss‚0.01094139718853606 2loss/FBt2/assignment_loss‚0.008303072530791093 _runtime‚7542.509212622 _step‚353áø ”Ñ‚(:‰ è§ëÒ¢Ü•È gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚157.752' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.5504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01L{õíÒ‚(:â è§ëÒŽ­ÓÉ network.sent‚ 8.21349e+06$ disk./.usageGB‚602.3784217834473 disk.sda3.in‚ 0.47265625) disk./.usagePercent‚40.42612131504803 disk.rbd1.in‚0 disk.sda.out‚ 339.58203125# disk.nvme0c0n1.in‚ 9718.02734375' disk.nvme0c0n1.out‚65802.2158203125 disk.dm-0.in‚ 9718.02734375 disk.rbd0.in‚0 disk.dm-1.out‚ 339.58203125 disk.rbd1.out‚ 20.19921875 disk.sda1.in‚0 disk.sda3.out‚ 339.58203125 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625! disk.nvme0n1.in‚ 9718.02734375 disk.sda1.out‚0$ memory_percent‚91.83154106140137 disk.sda2.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚65802.2158203125( proc.memory.availableMB‚ 1338.3203125 cpu‚25.513880508365418 disk.drbd2165.out‚0" disk.dm-0.out‚65802.2158203125 network.recv‚ 1.904245e+06 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2930.12109375* proc.memory.percent‚17.884039878845215ÃJúUÓ‚(:” ÷§ëÒ÷Ü¿Ç gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllŒR’8Jocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚267.174' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.434799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´æÔ‚(:Û ÷§ëÒÙ˜ŸÉ% disk.nvme0n1.out‚68531.0673828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.dm-1.out‚ 339.796875* proc.memory.percent‚17.884039878845215 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 9718.9921875 disk.nvme0n1.in‚ 9718.9921875 disk.sda2.in‚0 disk.sda.out‚ 339.796875# memory_percent‚91.8311357498169 cpu‚25.513592236034782 disk.drbd2165.out‚0 network.sent‚ 8.276289e+06 network.recv‚ 1.908939e+06* disk./.usagePercent‚40.591387234784385 disk.rbd0.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚604.8410034179688 disk.dm-0.in‚ 9718.9921875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2930.12109375' disk.nvme0c0n1.out‚68531.0673828125" disk.dm-0.out‚68529.8916015625 disk.dm-1.in‚ 0.47265625) proc.memory.availableMB‚ 1338.38671875 disk.rbd1.out‚ 20.19921875 disk.sda.in‚ 0.47265625 disk.sda3.out‚ 339.796875 disk.sda2.out‚0 disk.drbd2165.in‚0yâ4”Õ‚(:‰ †¨ëÒí¾Ç gpu.0.gpu‚81.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚268.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UÎóÝÖ‚(:Ò †¨ëÒÇ«§É proc.cpu.threads‚17* disk./.usagePercent‚40.765348706428746 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 9718.9921875 disk.dm-0.out‚ 70454.2109375 disk.rbd0.in‚0# memory_percent‚91.8328046798706 network.sent‚ 8.281267e+06$ disk./.usageGB‚607.4331550598145 disk.dm-1.out‚ 340.265625" disk.nvme0n1.out‚ 70454.2109375 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2930.12109375 disk.sda2.in‚0 disk.sda3.out‚ 340.265625 disk.sda.out‚ 340.265625 disk.drbd2165.out‚0 disk.dm-0.in‚ 9718.9921875 disk.nvme0n1.in‚ 9718.9921875 disk.sda3.in‚ 0.47265625 network.recv‚ 1.912418e+06 disk.sda.in‚ 0.47265625$ disk.nvme0c0n1.out‚ 70454.2109375 cpu‚25.513301845471197* proc.memory.percent‚17.884039878845215) proc.memory.availableMB‚ 1338.11328125 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 20.26171875 disk.drbd2165.in‚0]ñKx”ׂ(:‰ •¨ëÒëí•È gpu.0.gpu‚85.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚280.902' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#”£%ÚØ‚(:Ï •¨ëÒžÔÕÉ proc.cpu.threads‚17 disk.rbd1.in‚0$ disk.nvme0c0n1.out‚ 73895.7734375' proc.memory.availableMB‚ 1338.359375* proc.memory.percent‚17.884039878845215$ disk./.usageGB‚610.1238059997559 disk.nvme0c0n1.in‚ 9820.21875 network.sent‚ 8.28635e+06 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.47265625 disk.dm-0.in‚ 9820.21875# proc.memory.rssMB‚ 2930.12109375 cpu‚25.513043252805144 disk.sda.out‚ 340.69921875 disk.drbd2165.out‚0 disk.dm-0.out‚ 73895.7734375" disk.nvme0n1.out‚ 73895.7734375 disk.nvme0n1.in‚ 9820.21875$ memory_percent‚91.83130264282227) disk./.usagePercent‚40.94592054861472 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 340.69921875 disk.rbd0.in‚0 disk.dm-1.out‚ 340.69921875 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0 network.recv‚ 1.916099e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 20.26171875%$Àå”Ù‚(:‰ ¤¨ëÒÁÈ—Ç gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚330.673' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.1346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º9‚dãÚ‚(:Ø ¤¨ëÒ·ÆîÉ disk.rbd0.out‚0 network.recv‚ 1.919474e+06$ disk./.usageGB‚612.6324996948242 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 20.33984375 disk.nvme0c0n1.in‚ 9820.21875( disk.nvme0c0n1.out‚76340.29931640625 disk.dm-1.in‚ 0.47265625 disk.sda.out‚ 340.9609375' proc.memory.availableMB‚ 1338.609375# proc.memory.rssMB‚ 2930.12109375 disk.rbd1.in‚0 cpu‚25.51275424479171 disk.drbd2165.in‚0 disk.dm-0.in‚ 9820.21875 network.sent‚ 8.291462e+06 disk.nvme0n1.in‚ 9820.21875 disk.sda1.in‚0 disk.sda3.out‚ 340.9609375 proc.cpu.threads‚17) disk./.usagePercent‚41.11428108742498& disk.nvme0n1.out‚76340.29931640625 disk.sda3.in‚ 0.47265625 disk.sda2.in‚0 disk.sda.in‚ 0.47265625$ memory_percent‚91.82977676391602 disk.dm-1.out‚ 340.9609375 disk.drbd2165.out‚0# disk.dm-0.out‚76339.92431640625* proc.memory.percent‚17.884039878845215¶€Ê\”Û‚(:‰ ³¨ëÒÓþŒÇ gpu.0.gpu‚66.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚331.878' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.3756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!Ý¿ßÜ‚(:Ô ³¨ëÒÙöÚÉ disk.dm-1.in‚ 0.47265625" disk.nvme0n1.out‚ 81893.1953125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 9821.19921875 disk.sda1.in‚0* proc.memory.percent‚17.884039878845215 proc.cpu.threads‚17 network.recv‚ 1.92329e+06) proc.memory.availableMB‚ 1338.35546875 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 20.33984375 disk.sda3.in‚ 0.47265625$ memory_percent‚91.83132648468018 cpu‚25.512468700133738 disk.sda3.out‚ 341.390625) disk./.usagePercent‚41.42565537109542 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 9821.19921875 disk.rbd1.in‚0 disk.sda.out‚ 341.390625$ disk.nvme0c0n1.out‚ 81893.1953125 disk.sda1.out‚0 network.sent‚ 8.296517e+06$ disk./.usageGB‚617.2722015380859 disk.dm-1.out‚ 341.390625# disk.nvme0c0n1.in‚ 9821.19921875 disk.dm-0.out‚ 81893.1953125# proc.memory.rssMB‚ 2930.12109375ôˆ%22Ý- _wandbruntime‚7624  _runtime‚7624Éo’þÅÞ¿ & lr-AdamW/pg1‚0.0004357400766810545 & lr-AdamW/pg2‚0.0004357400766810545 trainer/global_step‚4349 $ _timestamp‚1.784337463958452e+09 _runtime‚ 7624.54353771 _step‚354ÊþgÊßÄ & lr-AdamW/pg1‚0.0004357400766810545 & lr-AdamW/pg2‚0.0004357400766810545 trainer/global_step‚4349 $ _timestamp‚1.784337463958452e+09 _runtime‚ 7624.54353771 _step‚354â°Ø>2à- _wandbruntime‚7624  _runtime‚7624æ11 þ'áøO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12353542 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37626564 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4080214934298 trainer/global_step‚4349 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37636537 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1247670466404352 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37626564 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1251211527729946 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12353810 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1247670492577792 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1251759045189120 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚337739115327488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74647089 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37020531 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚337738747236352 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1251212897758236 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23508971 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 $ _timestamp‚1.784337463959682e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37020525 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4081559534620 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1247131338223616 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37020531 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11155429 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37020525 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11155434 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _step‚355 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37636537 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343010517414400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚343010112689664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4087207221760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1251757673626112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1247131312795648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11155434 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _step‚355 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37636537 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343010517414400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚343010112689664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4087207221760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1251757673626112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1247131312795648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37626564 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1251211527729946 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12353810 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268ãb!f2ã- _wandbruntime‚7625  _runtime‚7625œ¿Wåcä‚N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37028731 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37028731 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74663773 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12356279 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5272551728640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4088132257280 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1252037970961920 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 trainer/global_step‚4349 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _step‚356 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37645029 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1247948493307904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4088132257280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74673766 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1247409220001792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚337816601735168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4082483000978 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74663773 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1247409194573824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4089477654016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12356538 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚343089190090240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚7625.962493868 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1247948467134464 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37028737 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37635042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11157977 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37028737 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4089477654016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23514515 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11157972 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚343088785372672 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚337816233644032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23514251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23514251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12356279 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5272551728640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4088132257280 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1252037970961920 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 trainer/global_step‚4349 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _step‚356 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37645029 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1247948493307904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4088132257280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74673766 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1247409220001792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0äøç*2æ- _wandbruntime‚7626  _runtime‚7626äüLàç€ 1loss/FBt1/assignment_loss‚0.02230188765241503 1loss/FBt2/assignment_loss‚0.04040704205814448 1loss/SRqqt1/detection_loss‚0.7517212281042399 0loss/SRqqt2/detection_loss‚0.730404828024513 _step‚357 0loss/FRt2/assignment_loss‚0.9109934042381352 2loss/SRqqt2/assignment_loss‚0.6057909733276289 /loss/FRt2/detection_loss‚1.8551411431864502 /loss/FBt2/detection_loss‚0.7894418604139771 trainer/global_step‚4349 /loss/FBt1/detection_loss‚0.4162937920359582 epoch‚10 _runtime‚7626.034778151 0loss/FRt1/assignment_loss‚1.7204096171675536 1loss/SRqqt1/assignment_loss‚0.502743321565582 /loss/FRt1/detection_loss‚0.5312806338480205 %loss/total_loss‚8.876929731622617 % _timestamp‚1.7843374654496324e+09´±Üº è… 0loss/FRt2/assignment_loss‚0.9109934042381352 2loss/SRqqt2/assignment_loss‚0.6057909733276289 /loss/FRt2/detection_loss‚1.8551411431864502 /loss/FBt2/detection_loss‚0.7894418604139771 trainer/global_step‚4349 /loss/FBt1/detection_loss‚0.4162937920359582 epoch‚10 _runtime‚7626.034778151 0loss/FRt1/assignment_loss‚1.7204096171675536 1loss/SRqqt1/assignment_loss‚0.502743321565582 /loss/FRt1/detection_loss‚0.5312806338480205 %loss/total_loss‚8.876929731622617 % _timestamp‚1.7843374654496324e+09 1loss/FBt1/assignment_loss‚0.02230188765241503 1loss/FBt2/assignment_loss‚0.04040704205814448 1loss/SRqqt1/detection_loss‚0.7517212281042399 0loss/SRqqt2/detection_loss‚0.730404828024513 _step‚357åž»}”é‚(:‰ ¨ëÒ®ÕÈÉ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚325.989' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.1978 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l§dæê‚(:Û ¨ëÒ áÞÊ disk.drbd2165.in‚0) proc.memory.availableMB‚ 1338.37109375 disk.sda1.out‚0 disk.dm-0.in‚ 9822.03125% disk.nvme0n1.out‚82755.0517578125) disk./.usagePercent‚41.53648743285639 disk.sda2.in‚0 disk.dm-1.out‚ 341.96484375 disk.rbd1.in‚0' disk.nvme0c0n1.out‚82755.0517578125 network.sent‚ 8.314283e+06 disk.sda3.in‚ 0.47265625 disk.sda2.out‚0" disk.dm-0.out‚82755.0517578125 cpu‚25.512175974428814$ disk./.usageGB‚618.9236793518066 disk.rbd0.in‚0 disk.rbd1.out‚ 20.46484375 disk.nvme0n1.in‚ 9822.03125# proc.memory.rssMB‚ 2930.12109375 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 9822.03125* proc.memory.percent‚17.884039878845215 disk.sda3.out‚ 341.96484375 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625$ memory_percent‚91.83123111724854 proc.cpu.threads‚17 network.recv‚ 1.926579e+06 disk.sda.in‚ 0.47265625 disk.sda.out‚ 341.964843757Õn­”ë‚(:‰ ѨëÒì”Ç gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚270.152' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0304 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@åæì‚(:Û ѨëÒÙ‘É$ disk.nvme0n1.out‚85063.880859375* proc.memory.percent‚17.884039878845215 disk.nvme0n1.in‚ 9932.0546875 disk.rbd1.out‚ 20.46484375 disk.sda3.out‚ 342.30078125 disk.rbd0.in‚0 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 342.30078125 disk.sda.in‚ 0.47265625" disk.nvme0c0n1.in‚ 9932.0546875 network.recv‚ 1.931356e+06 disk.sda2.in‚0 disk.dm-0.in‚ 9932.0546875$ memory_percent‚91.83204174041748# proc.memory.rssMB‚ 2930.12109375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd0.out‚0! disk.dm-0.out‚85063.880859375 cpu‚25.5118375636184$ disk./.usageGB‚620.0566940307617 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda.out‚ 342.30078125 disk.drbd2165.in‚0 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1338.23828125 disk.sda3.in‚ 0.47265625 disk.rbd1.in‚0& disk.nvme0c0n1.out‚85063.880859375 network.sent‚ 8.364182e+06( disk./.usagePercent‚41.6125249986237 è!í‚(:’ à¨ëÒË˱Ç gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V)êìî‚(:á à¨ëÒᬸÉ disk.sda1.out‚0 disk.sda.out‚ 342.6953125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0* disk./.usagePercent‚41.748574570529954 disk.sda3.in‚ 0.47265625 network.sent‚ 8.369152e+06 disk.sda1.in‚0 disk.sda3.out‚ 342.6953125 disk.rbd0.in‚0 disk.rbd1.out‚ 20.54296875" disk.nvme0c0n1.in‚ 9955.7109375$ memory_percent‚91.83201789855957* proc.memory.percent‚17.884039878845215 disk.nvme0n1.in‚ 9955.7109375& disk.nvme0n1.out‚85107.17431640625 disk.sda2.in‚0( disk.nvme0c0n1.out‚85107.17431640625 network.recv‚ 1.934688e+06$ disk./.usageGB‚622.0839309692383 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 342.6953125 disk.dm-0.in‚ 9955.7109375# disk.dm-0.out‚85107.17431640625 cpu‚25.511682307199887 disk.rbd1.in‚0# proc.memory.rssMB‚ 2930.12109375 proc.cpu.threads‚17 disk.sda.in‚ 0.47265625( proc.memory.availableMB‚ 1338.2421875œfFéŸï‚(:” ï¨ëÒ ñ›Ç gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.164599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø.ŽÄñð‚(:æ ï¨ëÒÝÂåÉ) disk./.usagePercent‚41.78214458248735 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0* proc.memory.percent‚17.884039878845215 disk.sda2.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚622.5841484069824) proc.memory.availableMB‚ 1338.00390625# proc.memory.rssMB‚ 2930.12109375" disk.nvme0n1.in‚13042.92578125 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.47265625( disk.nvme0c0n1.out‚87720.45654296875 disk.sda1.in‚0 disk.dm-1.out‚ 343.6328125$ memory_percent‚91.83347225189209 disk.dm-0.in‚13042.92578125# disk.dm-0.out‚87720.45654296875 network.recv‚ 1.938343e+06 disk.rbd1.out‚ 20.54296875 disk.sda3.in‚ 0.47265625 disk.sda2.out‚0 cpu‚25.51133272211678 network.sent‚ 8.374123e+06 disk.sda.out‚ 343.6328125$ disk.nvme0c0n1.in‚13042.92578125& disk.nvme0n1.out‚87720.45654296875 disk.sda1.out‚0 disk.sda3.out‚ 343.6328125 disk.rbd0.out‚0h«¸F”ñ‚(:‰ þ¨ëÒ÷š¶Ç gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0» ëò‚(:à þ¨ëÒÓ‚—É) disk./.usagePercent‚41.78214560651748' disk.nvme0c0n1.out‚87726.5107421875 network.sent‚ 8.379151e+06 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚91.83347225189209) proc.memory.availableMB‚ 1338.00390625$ disk./.usageGB‚622.5841636657715 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 344.02734375 disk.rbd1.out‚ 20.60546875 disk.sda1.in‚0 disk.sda3.out‚ 344.02734375 disk.dm-0.in‚ 22515.71875 disk.rbd0.out‚0* proc.memory.percent‚17.884039878845215 proc.cpu.threads‚17 disk.sda.in‚ 0.47265625" disk.nvme0c0n1.in‚ 22515.703125% disk.nvme0n1.out‚87726.5107421875# proc.memory.rssMB‚ 2930.12109375 cpu‚25.511088209639194 disk.sda.out‚ 344.02734375 network.recv‚ 1.941898e+06 disk.sda2.in‚0 disk.rbd1.in‚0" disk.dm-0.out‚87726.5107421875 disk.sda3.in‚ 0.47265625 disk.nvme0n1.in‚ 22515.703125 disk.sda1.out‚0 disk.rbd0.in‚0q©fö2ó- _wandbruntime‚7708  _runtime‚7708ìÞÜ*ÇôÁ & lr-AdamW/pg1‚0.0004345901985814935 & lr-AdamW/pg2‚0.0004345901985814935 trainer/global_step‚4399 % _timestamp‚1.7843375480425773e+09 _runtime‚7708.627604102 _step‚358¼â’ÌõÆ % _timestamp‚1.7843375480425773e+09 _runtime‚7708.627604102 _step‚358 & lr-AdamW/pg1‚0.0004345901985814935 & lr-AdamW/pg2‚0.0004345901985814935 trainer/global_step‚4399æí0Í)2ö- _wandbruntime‚7708  _runtime‚7708ßɉ—ü÷ùO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4133523152384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38062289 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚341613435711488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75493120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11282579 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37430831 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1265705397625344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37430825 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚341613067620352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1265704026062336 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4134868541952 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1261570529083392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1261570502909952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23774010 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚346944061033984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1261570502909952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38062289 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1261570529083392 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4133523152384 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1265704026062336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1265153223780144 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1261025427132416 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12491431 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5330590786048 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640YØ<1(øþO WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1265705397625344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 trainer/global_step‚4399 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37430831 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11282584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1261025401704448 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37430825 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38052316 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1265704026062336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1265153223780144 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1261025427132416 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12491431 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5330590786048 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4133523152384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38062289 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚341613435711488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75493120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11282579 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37430831 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1265705397625344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37430825 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚341613067620352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1265704026062336 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4134868541952 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1261570529083392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1261570502909952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23774010 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚346944061033984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1261570502909952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38062289 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1261570529083392 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4133523152384 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75499825 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1261848529813504 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37439031 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23779292 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1261303283482624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4127375513652 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1265982951827968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚347022327453184 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11285122 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843375494625592e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11285127 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚341690922119168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5331810051584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75499825 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚4399 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38060794 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75509818 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1261303308910592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37439037 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚7710.048126281 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1265984323398144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38070781 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12494422 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1265430658996276 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚347022732170752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚341690554028032 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37439031 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4135793584640 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1265984323398144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12494170 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1265432029024678 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1261848529813504 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4134448187904 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38070781ˆQÙW#ýˆN TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚7710.048126281 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1265984323398144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38070781 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12494422 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1265430658996276 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚347022732170752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚341690554028032 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37439031 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4135793584640 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1265984323398144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12494170 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1265432029024678 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1261848529813504 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4134448187904 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38070781 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1261848503640064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23779549 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75509818 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75499825 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1261848529813504 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37439031 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23779292 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1261303283482624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4127375513652 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1265982951827968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚347022327453184 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11285122 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843375494625592e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11285127 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚341690922119168 aODeviceStatsMonitor.on_train_batüÒè~ ch_end/inactive_split_bytes.small_pool.allocated‚ 5331810051584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75499825 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚4399 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38060794 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75509818 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1261303308910592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37439037謱ë2þ- _wandbruntime‚7710  _runtime‚7710&˜Iÿ ÿ„ 0loss/FRt1/assignment_loss‚1.7221313942743426 1loss/SRqqt1/detection_loss‚0.7670524521119814 %loss/total_loss‚9.067196929900785 epoch‚10 _runtime‚7710.114156292 2loss/SRqqt1/assignment_loss‚0.5197274268087567 2loss/FBt2/assignment_loss‚0.007284161534892222 /loss/FRt2/detection_loss‚1.6696050111063787 trainer/global_step‚4399 0loss/FRt2/assignment_loss‚0.8010883321220749 1loss/SRqqt2/detection_loss‚0.6918876898174252 /loss/FBt1/detection_loss‚0.8555444819117334 _step‚361 2loss/SRqqt2/assignment_loss‚0.5722799634393855 2loss/FBt1/assignment_loss‚0.007680995325961192 /loss/FRt1/detection_loss‚0.5878490765838864 /loss/FBt2/detection_loss‚0.8650659448639663 % _timestamp‚1.7843375495290885e+09F!¿Ž€‰ %loss/total_loss‚9.067196929900785 epoch‚10 _runtime‚7710.114156292 2loss/SRqqt1/assignment_loss‚0.5197274268087567 2loss/FBt2/assignment_loss‚0.007284161534892222 /loss/FRt2/detection_loss‚1.6696050111063787 trainer/global_step‚4399 0loss/FRt2/assignment_loss‚0.8010883321220749 1loss/SRqqt2/detection_loss‚0.6918876898174252 /loss/FBt1/detection_loss‚0.8555444819117334 _step‚361 2loss/SRqqt2/assignment_loss‚0.5722799634393855 2loss/FBt1/assignment_loss‚0.007680995325961192 /loss/FRt1/detection_loss‚0.5878490765838864 /loss/FBt2/detection_loss‚0.8650659448639663 % _timestamp‚1.7843375495290885e+09 0loss/FRt1/assignment_loss‚1.7221313942743426 1loss/SRqqt1/detection_loss‚0.7670524521119814é9ªÿH‚(:’ œ©ëÒ‡ÆÇ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.415' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.083000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A® è‚‚(:Ý œ©ëҋψÊ) disk./.usagePercent‚41.78322519028937 disk.sda1.out‚0 disk.sda3.out‚ 344.73046875 disk.rbd1.in‚0 disk.drbd2165.out‚0 network.recv‚ 1.949797e+06 disk.rbd1.out‚ 20.68359375# disk.dm-0.out‚88079.82568359375$ memory_percent‚91.84679985046387 disk.nvme0c0n1.in‚ 23336.9375( disk.nvme0c0n1.out‚88079.82568359375 disk.sda1.in‚0 cpu‚25.515329464305722 proc.cpu.threads‚17( proc.memory.availableMB‚ 1335.8203125 disk.sda3.in‚ 0.47265625 disk.dm-1.in‚ 0.47265625 disk.sda.in‚ 0.47265625 disk.nvme0n1.in‚ 23336.9375& disk.nvme0n1.out‚88079.82568359375 disk.rbd0.out‚0 disk.sda.out‚ 344.73046875* proc.memory.percent‚17.896342277526855 network.sent‚ 8.436489e+06 disk.dm-1.out‚ 344.73046875 disk.dm-0.in‚ 23336.9375# proc.memory.rssMB‚ 2932.13671875 disk.sda2.out‚0$ disk./.usageGB‚622.6002502441406 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2165.in‚0|9æ2ƒ- _wandbruntime‚7738  _runtime‚7738…NR¡=„›{ 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5373402833938599 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1944444477558136 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9856237769126892 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3389398157596588 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9307677149772644 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3563278615474701 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8771731853485107 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.49568411707878113 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.47077345848083496 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10221223533153534 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963369965553284 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4061743915081024 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.4965709447860718 +"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.75 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6598817706108093 epoch‚10 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5633063316345215 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2753303349018097 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9920010566711426 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8877362012863159 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9165121912956238 particle/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.18502533435821533 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.601455569267273 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5525000095367432 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3840073049068451 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5898438692092896 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.41558870673179626 particle/accuracy_1_of_1‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7245911955833435 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1944444477558136 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.994841456413269 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3435174524784088 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.3412078619003296 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.25918954610824585 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971345067024231 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.31574922800064087 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.34324899315834045 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.35910457372665405 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9978081583976746 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8879063129425049 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ,jet/accuracy_4_of_4‚0.013899499550461769 /particle/accuracy_4_of_4‚0.9235150218009949 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2930234670639038 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9941871166229248 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.4918007254600525 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9928986430168152 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3379436731338501 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.29592427611351013 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27542123198509216 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6442263722419739 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3404484987258911 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11658768355846405 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.39815202355384827 +jet/accuracy_1_of_1‚0.25542178750038147 jet/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3506838083267212 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 +jet/accuracy_3_of_3‚0.07231093943119049 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4677543044090271 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_1_of_5‚1 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.36515021324157715 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.334216445684433 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6485798954963684 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.3300170600414276 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2786455750465393 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.38097885251045227 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9935483336448669 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6060413718223572 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26869457960128784 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11695975065231323 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.3337635397911072 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 &particle/f_score‚0.624406099319458 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.08126702159643173 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942063689231873 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚4399 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3888888955116272 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.036471519619226456 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.65625 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.33120694756507874 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 'particle/accuracy‚0.844451367855072 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9808043241500854 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2929319441318512 particle/accuracy_3_of_4‚1 jet/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9872716069221497 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4722222089767456 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6598817706108093 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8611111044883728 particle/accuracy_4_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 particle/accuracy_5_of_5‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02594190463423729 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_1_of_2‚1 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.731539249420166 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20851439237594604 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.488923579454422 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28139153122901917 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9712474346160889 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.17965976893901825 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5075738430023193 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5805545449256897 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9970848560333252 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.34329456090927124 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.29265308380126953 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.052617624402046204 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3891526758670807 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9781746864318848 7validation_average_jet_accuracy‚0.30650824308395386 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9802122712135315 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9140046238899231 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.23850266635417938 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3750000298023224 particle/accuracy_3_of_5‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.35955873131752014 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8699532747268677 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5270833373069763 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.573283851146698 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5145381093025208 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8934062123298645 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.34631407260894775 *jet/accuracy_1_of_4‚0.9627428650856018 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5246501564979553 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2831557095050812 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5611165165901184 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.4932430386543274 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5299921631813049 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.4933489263057709 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 +particle/specificity‚0.9107415080070496 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9970848560333252 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2831557095050812 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8490174412727356 _step‚362 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2737376391887665 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.33120694756507874 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7749810814857483 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.988099217414856 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6024407744407654 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 +particle/sensitivity‚0.6020811796188354 % _timestamp‚1.7843375775550013e+09 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23148055374622345 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4115285873413086 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21964679658412933 *jet/accuracy_2_of_4‚0.7849752306938171 jet/accuracy_4_of_5‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2107178419828415 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9867331981658936 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.986925482749939 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8820483684539795 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2712574005126953 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.995988130569458 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.294155091047287 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5520479083061218 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 jet/accuracy_2_of_5‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6293425559997559 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.08122161775827408 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚7738.140888702 )loss/val_total_loss‚8.403705975882193 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8529956340789795 ,"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6491317749023438 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.23850266635417938 particle/accuracy_2_of_2‚1 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1983572244644165 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0797199159860611 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9922719597816467 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46427273750305176 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06017061695456505 +"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.75 *jet/accuracy_2_of_3‚0.4968782961368561 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5482916831970215 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1l Ú^2… { A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.488923579454422 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28139153122901917 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9712474346160889 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.17965976893901825 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5075738430023193 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5805545449256897 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9970848560333252 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.34329456090927124 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.29265308380126953 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.052617624402046204 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3891526758670807 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9781746864318848 7validation_average_jet_accuracy‚0.30650824308395386 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9802122712135315 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9140046238899231 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.23850266635417938 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3750000298023224 particle/accuracy_3_of_5‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.35955873131752014 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8699532747268677 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5270833373069763 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.573283851146698 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5145381093025208 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8934062123298645 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.34631407260894775 *jet/accuracy_1_of_4‚0.9627428650856018 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5246501564979553 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2831557095050812 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5611165165901184 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.4932430386543274 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5299921631813049 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.4933489263057709 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 +particle/specificity‚0.9107415080070496 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9970848560333252 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2831557095050812 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8490174412727356 _step‚362 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2737376391887665 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.33120694756507874 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7749810814857483 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.988099217414856 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6024407744407654 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 +particle/sensitivity‚0.6020811796188354 % _timestamp‚1.7843375775550013e+09 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23148055374622345 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4115285873413086 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21964679658412933 *jet/accuracy_2_of_4‚0.7849752306938171 jet/accuracy_4_of_5‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2107178419828415 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9867331981658936 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.986925482749939 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8820483684539795 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2712574005126953 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.995988130569458 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.294155091047287 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5520479083061218 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 jet/accuracy_2_of_5‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6293425559997559 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.08122161775827408 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚7738.140888702 )loss/val_total_loss‚8.403705975882193 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8529956340789795 ,"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6491317749023438 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.23850266635417938 particle/accuracy_2_of_2‚1 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1983572244644165 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0797199159860611 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9922719597816467 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46427273750305176 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06017061695456505 +"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.75 *jet/accuracy_2_of_3‚0.4968782961368561 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5482916831970215 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5373402833938599 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1944444477558136 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9856237769126892 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3389398157596588 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9307677149772644 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3563278615474701 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8771731853485107 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.49568411707878113 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.47077345848083496 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10221223533153534 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963369965553284 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4061743915081024 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.4965709447860718 +"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.75 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6598817706108093 epoch‚10 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5633063316345215 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2753303349018097 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9920010566711426 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8877362012863159 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9165121912956238 particle/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.18502533435821533 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.601455569267273 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5525000095367432 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3840073049068451 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5898438692092896 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.41558870673179626 particle/accuracy_1_of_1‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7245911955833435 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1944444477558136 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.994841456413269 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3435174524784088 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.3412078619003296 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.25918954610824585 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971345067024231 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.31574922800064087 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.34324899315834045 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.35910457372665405 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9978081583976746 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8879063129425049 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ,jet/accuracy_4_of_4‚0.013899499550461769 /particle/accuracy_4_of_4‚0.9235150218009949 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2930234670639038 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9941871166229248 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.4918007254600525 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9928986430168152 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3379436731338501 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.29592427611351013 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27542123198509216 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6442263722419739 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3404484987258911 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11658768355846405 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.39815202355384827 +jet/accuracy_1_of_1‚0.25542178750038147 jet/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3506838083267212 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 +jet/accuracy_3_of_3‚0.07231093943119049 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4677543044090271 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_1_of_5‚1 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.36515021324157715 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.334216445684433 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6485798954963684 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.3300170600414276 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2786455750465393 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.žZwr‘ 0008370535797439516 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.38097885251045227 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9935483336448669 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6060413718223572 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26869457960128784 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11695975065231323 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.3337635397911072 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 &particle/f_score‚0.624406099319458 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.08126702159643173 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942063689231873 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚4399 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3888888955116272 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.036471519619226456 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.65625 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.33120694756507874 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 'particle/accuracy‚0.844451367855072 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9808043241500854 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2929319441318512 particle/accuracy_3_of_4‚1 jet/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9872716069221497 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4722222089767456 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6598817706108093 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8611111044883728 particle/accuracy_4_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 particle/accuracy_5_of_5‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02594190463423729 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_1_of_2‚1 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.731539249420166 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20851439237594604 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1êÌxâÔ†‚(:‰ «©ëÒ¹ÍÇÇ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚286.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.2526 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07Ú"~㇂(:Ø «©ëÒµèçÉ disk.drbd2165.in‚0$ memory_percent‚91.87672138214111) proc.memory.availableMB‚ 1330.91796875 disk.sda2.in‚0) disk./.usagePercent‚41.79431134055212 disk.nvme0n1.in‚ 23336.9375& disk.nvme0n1.out‚88331.38427734375 disk.sda1.out‚0 disk.dm-1.in‚ 0.47265625 network.sent‚ 8.488802e+06 disk.sda1.in‚0 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda.out‚ 345.078125 cpu‚25.519808858343737 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 20.68359375( disk.nvme0c0n1.out‚88331.38427734375$ disk./.usageGB‚622.7654418945312 disk.sda3.in‚ 0.47265625 disk.dm-0.in‚ 23336.9375# disk.dm-0.out‚88331.38427734375# proc.memory.rssMB‚ 2937.41796875 disk.dm-1.out‚ 345.078125 disk.nvme0c0n1.in‚ 23336.9375* proc.memory.percent‚17.928576469421387 network.recv‚ 1.954153e+06 disk.sda3.out‚ 345.078125 disk.rbd1.in‚0qH”ˆ‚(:‰ º©ëÒ°š¸Ç gpu.0.gpu‚82.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚118.804' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üöN?(:ä º©ëÒ»ÒÍÊ disk.rbd0.out‚0" disk.nvme0n1.in‚23337.90234375 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 345.4140625 disk.rbd1.out‚ 20.74609375 disk.drbd2165.in‚0 disk.sda3.in‚ 0.47265625 disk.sda2.out‚0$ disk.nvme0c0n1.in‚23337.90234375' disk.nvme0c0n1.out‚88339.3173828125) disk./.usagePercent‚41.80602522128882 disk.dm-1.in‚ 0.47265625 disk.sda.out‚ 345.4140625 disk.drbd2165.out‚0 network.recv‚ 1.958117e+06 disk.sda3.out‚ 345.4140625 disk.rbd1.in‚0* proc.memory.percent‚17.928576469421387 proc.cpu.threads‚17 disk.dm-0.in‚23337.90234375" disk.dm-0.out‚88339.3173828125 network.sent‚ 8.546226e+06% disk.nvme0n1.out‚88339.3173828125 disk.sda2.in‚0 disk.sda.in‚ 0.47265625 cpu‚25.520247217513962$ disk./.usageGB‚622.9399871826172 disk.sda1.out‚0) proc.memory.availableMB‚ 1330.91015625# proc.memory.rssMB‚ 2937.41796875$ memory_percent‚91.87676906585693£Ó@$•Š‚(:Š É©ëÒëè«Ç gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚335.402' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.0804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·0Çç‹‚(:Ü É©ëÒ½ªßÉ) proc.memory.availableMB‚ 1330.42578125$ disk./.usageGB‚623.1699905395508# disk.dm-0.out‚88660.96923828125 proc.cpu.threads‚17 disk.dm-1.in‚ 0.47265625 disk.sda.out‚ 345.890625 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625( disk.nvme0c0n1.out‚88660.96923828125 disk.nvme0n1.in‚ 23338.734375 disk.sda1.in‚0# proc.memory.rssMB‚ 2937.41796875 network.sent‚ 8.551194e+06 disk.rbd1.out‚ 20.74609375 disk.drbd2165.out‚0 disk.dm-0.in‚ 23338.734375& disk.nvme0n1.out‚88660.96923828125 disk.sda3.out‚ 345.890625 cpu‚25.520631034644538 disk.rbd0.in‚0 disk.sda2.in‚0# memory_percent‚91.8797254562378* proc.memory.percent‚17.928576469421387 network.recv‚ 1.961427e+06( disk./.usagePercent‚41.8214609395583 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 345.890625 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625" disk.nvme0c0n1.in‚ 23338.734375?C(džŒ‚(:“ Ø©ëÒêÝ£Ç gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚264.47' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.894000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š?*Âñ‚(:æ Ø©ëÒ†è…É disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 346.37109375( proc.memory.availableMB‚ 1329.7265625& disk.nvme0n1.out‚88812.02001953125 disk.drbd2165.in‚0$ memory_percent‚91.88399314880371 network.sent‚ 8.55617e+06# disk.dm-0.out‚88812.02001953125" disk.nvme0n1.in‚23417.73046875 disk.sda3.out‚ 346.37109375 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 20.82421875 disk.drbd2165.out‚0 network.recv‚ 1.964574e+06) disk./.usagePercent‚41.83727990110654 disk.rbd0.out‚0 disk.sda.in‚ 0.47265625$ disk.nvme0c0n1.in‚23417.73046875( disk.nvme0c0n1.out‚88812.02001953125# proc.memory.rssMB‚ 2937.41796875 proc.cpu.threads‚17# disk./.usageGB‚623.405704498291 disk.dm-0.in‚23417.73046875 disk.sda1.out‚0* proc.memory.percent‚17.928576469421387 cpu‚25.52094829248905 disk.sda3.in‚ 0.47265625 disk.dm-1.in‚ 0.47265625 disk.sda.out‚ 346.37109375;US<ŸŽ‚(:” ç©ëÒ°£ÞÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚289.503' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.900600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ññöÓï‚(:ä ç©ëÒ°£É disk.dm-0.in‚ 23418.6953125) proc.memory.availableMB‚ 1329.69921875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0$ memory_percent‚91.88416004180908 proc.cpu.threads‚17 disk.sda3.in‚ 0.47265625 disk.dm-1.out‚ 346.88671875# disk.nvme0c0n1.in‚ 23418.6953125* proc.memory.percent‚17.928576469421387 disk.dm-1.in‚ 0.47265625 disk.sda.out‚ 346.88671875 disk.drbd2165.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 346.88671875# proc.memory.rssMB‚ 2937.41796875$ disk./.usageGB‚623.6359519958496 disk.sda.in‚ 0.47265625% disk.nvme0n1.out‚89094.0869140625 disk.sda1.out‚0 cpu‚25.52128998667256 network.recv‚ 1.968159e+06* disk./.usagePercent‚41.852732003858215 disk.sda2.out‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 23418.6953125 disk.rbd1.out‚ 20.82421875' disk.nvme0c0n1.out‚89094.0869140625" disk.dm-0.out‚89094.0869140625 disk.rbd0.in‚0 network.sent‚ 8.561185e+06 ƒüž‚(:“ ö©ëÒÿŠ®Ç gpu.0.gpu‚85.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚336.465' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.29299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fEժᑂ(:Ö ö©ëÒö·ÏÊ$ disk./.usageGB‚623.8713645935059 disk.sda.in‚ 0.47265625# disk.nvme0c0n1.in‚ 23418.6953125 disk.sda2.out‚0# proc.memory.rssMB‚ 2937.41796875 network.recv‚ 1.9716e+06" disk.dm-0.out‚89214.1220703125 disk.sda3.in‚ 0.47265625* proc.memory.percent‚17.928576469421387 disk.rbd1.in‚0' disk.nvme0c0n1.out‚89214.1220703125! disk.nvme0n1.in‚ 23418.6953125 disk.rbd0.out‚0 disk.dm-1.out‚347.3125$ memory_percent‚91.88425540924072 disk.drbd2165.in‚0% disk.nvme0n1.out‚89214.1220703125 disk.sda3.out‚347.3125 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 disk.dm-0.in‚ 23418.6953125 disk.rbd1.out‚ 20.88671875 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1329.68359375 proc.cpu.threads‚17) disk./.usagePercent‚41.86853074081125 disk.sda.out‚347.3125 disk.sda1.out‚0 network.sent‚ 8.566273e+06 disk.rbd0.in‚0 disk.sda1.in‚0 cpu‚25.521782220723292Bêð2’- _wandbruntime‚7818  _runtime‚7818J¼Ê·Ç“Á % _timestamp‚1.7843376574703517e+09 _runtime‚7818.056070882 _step‚363 & lr-AdamW/pg1‚0.0004334279017479087 & lr-AdamW/pg2‚0.0004334279017479087 trainer/global_step‚4449óþ0¸Ì”Æ & lr-AdamW/pg1‚0.0004334279017479087 & lr-AdamW/pg2‚0.0004334279017479087 trainer/global_step‚4449 % _timestamp‚1.7843376574703517e+09 _runtime‚7818.056070882 _step‚363ëkƒ€ê2•- _wandbruntime‚7818  _runtime‚7818Q5þ'–øO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚7818.056889016 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4179862892273 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4187027084288 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76520160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345535500251136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5397709346816 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1282925287866097 trainer/global_step‚4449 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4185681694720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 _step‚364 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4187027084288 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37973137 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38537050 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38547023 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11418647 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1283484202545152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37973137 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1279298520850432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1278745424973824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1282923917837807 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76510181 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12662076 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38537050 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1283485574108160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24080974 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1279298547023872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350933610128384 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350933209597952 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1278745399545856 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1283484202545152 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚7818.056889016 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4179862892273 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4187027084288 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76520160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345535500251136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5397709346816 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1282925287866097 trainer/global_step‚4449 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4185681694720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 _step‚364 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4187027084288 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37973137 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38537050 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38547023 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11418647 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1283484202545152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37973137 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1279298520850432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1278745424973824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1282923917837807 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76510181 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12662076 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38537050 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1283485574108160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24080974 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1279298547023872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350933610128384 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350933209597952 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1278745399545856 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1283484202545152 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ì9|¬†2˜- _wandbruntime‚7819  _runtime‚7819yɧ ™ƒN QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1279576547753984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24086023 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11421195 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38555515 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37981343 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4449 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345612986658816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1279576521580544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚351012299337728 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5398912155648 _runtime‚7819.459222816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843376588735313e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚351011898814464 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1279576547753984 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1283763128310784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4187952126976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37981343 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5398944587776 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37981337 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76536858 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37981337 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4187952126976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4186606730240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12665067 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1283764499880960 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345613354749952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚365 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1283764499880960 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76526865 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1279576521580544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38545528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38555515 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38545528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76536858 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76536858 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1279576547753984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24086023 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11421195 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38555515 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37981343 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4449 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345612986658816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1279576521580544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚351012299337728 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5398912155648 _runtime‚7819.459222816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843376588735313e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚351011898814464 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1279576547753984 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1283763128310784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4187952126976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37981343 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5398944587776 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37981337 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76536858 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37981337 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4187952126976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4186606730240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12665067 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1283764499880960 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345613354749952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚365 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1283764499880960 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76526865 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1279576521580544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38545528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38555515 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38545528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0íM 5Ì2›- _wandbruntime‚7819  _runtime‚7819òö¡¯ œ… 0loss/FRt1/assignment_loss‚1.6603608714145364 0loss/FRt2/assignment_loss‚0.7612840511969158 4loss/FBt2/assignment_loss‚0.00024604238791141596 .loss/FRt2/detection_loss‚1.550708775470654 /loss/FBt1/detection_loss‚0.5370142261773624 2loss/SRqqt2/assignment_loss‚0.5319695822277366 1loss/SRqqt1/detection_loss‚0.7430075749739884 1loss/SRqqt2/detection_loss‚0.6521478435838971 epoch‚11 trainer/global_step‚4449 % _timestamp‚1.7843376589538183e+09 _runtime‚7819.538609015 _step‚366 2loss/SRqqt1/assignment_loss‚0.4869928184596046 2loss/FBt1/assignment_loss‚0.043159405124040234 /loss/FBt2/detection_loss‚0.7746940034840788 /loss/FRt1/detection_loss‚0.5222856375179776 %loss/total_loss‚8.263870832018704åV&Š 0loss/FRt1/assignment_loss‚1.6603608714145364 0loss/FRt2/assignment_loss‚0.7612840511969158 4loss/FBt2/assignment_loss‚0.00024604238791141596 .loss/FRt2/detection_loss‚1.550708775470654 /loss/FBt1/detection_loss‚0.5370142261773624 2loss/SRqqt2/assignment_loss‚0.5319695822277366 1loss/SRqqt1/detection_loss‚0.7430075749739884 1loss/SRqqt2/detection_loss‚0.6521478435838971 epoch‚11 trainer/global_step‚4449 % _timestamp‚1.7843376589538183e+09 _runtime‚7819.538609015 _step‚366 2loss/SRqqt1/assignment_loss‚0.4869928184596046 2loss/FBt1/assignment_loss‚0.043159405124040234 /loss/FBt2/detection_loss‚0.7746940034840788 /loss/FRt1/detection_loss‚0.5222856375179776 %loss/total_loss‚8.263870832018704î¿—l”ž‚(:‰ …ªëÒœ†¨Ç gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚235.174' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.0348 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ûµ„o쟂(:á …ªëÒ²ÃæÉ disk.sda1.out‚0 disk.rbd0.out‚0 network.sent‚ 8.571258e+06 network.recv‚ 1.974939e+06" disk.nvme0n1.in‚23419.54296875 disk.rbd0.in‚0 disk.dm-1.out‚ 348.33984375% disk.nvme0c0n1.out‚89695.48046875 disk.dm-0.in‚23419.54296875# disk.nvme0n1.out‚89695.48046875 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 348.33984375 disk.sda2.out‚0 disk.rbd1.in‚0$ memory_percent‚91.88201427459717 disk.rbd1.out‚ 20.88671875 disk.sda.in‚ 0.47265625# proc.memory.rssMB‚ 2937.41796875 disk.dm-1.in‚ 0.47265625) proc.memory.availableMB‚ 1330.05078125 cpu‚25.522128045938256 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.out‚ 348.33984375 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚23419.54296875 disk.dm-0.out‚89695.48046875* proc.memory.percent‚17.928576469421387$ disk./.usageGB‚624.1841278076172 disk.sda1.in‚0 disk.drbd2165.in‚0) disk./.usagePercent‚41.88952054253608å²Ò9” ‚(:‰ ”ªëÒƒÍìÇ gpu.0.gpu‚88.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö-6jé¡‚(:Þ ”ªëÒýƒÄÉ" disk.nvme0c0n1.in‚ 23503.546875 disk.dm-0.in‚ 23503.546875 disk.sda1.in‚0 disk.sda3.in‚ 0.47265625 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625 cpu‚25.52243713677927 network.sent‚ 8.634085e+06) disk./.usagePercent‚41.90090954971183# disk./.usageGB‚624.353832244873 disk.sda2.out‚0$ memory_percent‚91.88656806945801* proc.memory.percent‚17.928576469421387 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625 disk.rbd1.out‚ 20.94921875( proc.memory.availableMB‚ 1329.3046875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 23503.546875 network.recv‚ 1.979607e+06( disk.nvme0c0n1.out‚89864.37353515625 disk.drbd2165.in‚0# disk.dm-0.out‚89864.37353515625& disk.nvme0n1.out‚89864.37353515625# proc.memory.rssMB‚ 2937.41796875 disk.sda1.out‚0 disk.sda3.out‚ 348.6171875 disk.dm-1.out‚ 348.6171875 disk.sda.out‚ 348.6171875 disk.drbd2165.out‚0 disk.sda2.in‚0A*µ“¢‚(:ˆ £ªëÒÞÕØÈ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚282.495' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.499 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇRÃÒ壂(:Ú £ªëÒîÔëÉ network.sent‚ 8.639062e+06 disk.nvme0n1.in‚ 23503.546875 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 20.96484375# disk.dm-0.out‚90055.63916015625 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625& disk.nvme0n1.out‚90055.63916015625 disk.dm-1.out‚ 349.15625 disk.sda.out‚ 349.15625( disk.nvme0c0n1.out‚90055.63916015625 network.recv‚ 1.983334e+06$ disk./.usageGB‚624.5176544189453 disk.rbd1.in‚0* proc.memory.percent‚17.928576469421387 cpu‚25.522836227629202) disk./.usagePercent‚41.91190379326978 disk.sda3.in‚ 0.47265625 disk.sda.in‚ 0.47265625 disk.drbd2165.in‚0 disk.dm-0.in‚ 23503.546875$ memory_percent‚91.88418388366699 disk.sda3.out‚ 349.15625 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 23503.546875 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0( proc.memory.availableMB‚ 1329.6953125# proc.memory.rssMB‚ 2937.41796875Q}ÍŸ¤‚(:” ²ªëÒÝÏ“É gpu.0.gpu‚83.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚179.596' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.919200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ"¹ä¥‚(:Ù ²ªëÒ™½ÎË! disk.dm-0.out‚90355.111328125 disk.sda3.out‚ 349.671875 disk.rbd0.in‚0$ memory_percent‚91.88423156738281* proc.memory.percent‚17.928576469421387 network.sent‚ 8.644036e+06 disk.dm-1.in‚ 0.47265625& disk.nvme0c0n1.out‚90355.111328125 disk.sda3.in‚ 0.47265625 disk.rbd0.out‚0) disk./.usagePercent‚41.92256548304075 cpu‚25.52328725710463 disk.dm-1.out‚ 349.671875 disk.sda2.out‚0% proc.memory.availableMB‚ 1329.6875# proc.memory.rssMB‚ 2937.41796875 disk.rbd1.out‚ 20.96484375 disk.dm-0.in‚23504.37890625 disk.rbd1.in‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚23504.37890625 disk.sda1.in‚0 disk.sda.out‚ 349.671875" disk.nvme0n1.in‚23504.37890625$ disk.nvme0n1.out‚90355.111328125 disk.sda1.out‚0$ disk./.usageGB‚624.6765213012695 disk.sda.in‚ 0.47265625 disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚ 1.986694e+06 disk.drbd2165.out‚0÷Ýp”¦‚(:‰ ÁªëÒ¼ú«Ç gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚180.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0842 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉŸòÕ짂(:á ÁªëÒ¾í…É* disk./.usagePercent‚41.927934473048836$ disk./.usageGB‚624.7565231323242# proc.memory.rssMB‚ 2937.41796875 proc.cpu.threads‚17 disk.sda.in‚ 0.47265625 disk.sda.out‚ 350.23046875 disk.dm-1.out‚ 350.23046875 disk.rbd1.out‚ 21.02734375# disk.dm-0.out‚90367.12451171875 disk.sda3.in‚ 0.47265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0* proc.memory.percent‚17.928576469421387 disk.nvme0n1.in‚ 23505.34375 disk.rbd0.in‚0$ memory_percent‚91.88573360443115 network.sent‚ 8.649001e+06 disk.drbd2165.in‚0& disk.nvme0n1.out‚90367.12451171875 disk.sda3.out‚ 350.23046875 cpu‚25.52363936190246) proc.memory.availableMB‚ 1329.44140625 disk.dm-1.in‚ 0.47265625! disk.nvme0c0n1.in‚ 23505.34375( disk.nvme0c0n1.out‚90367.12451171875 disk.dm-0.in‚ 23505.34375 disk.sda2.out‚0 network.recv‚ 1.989978e+0662¨- _wandbruntime‚7899  _runtime‚7899ØXµâÅ©¿ % lr-AdamW/pg2‚0.000432253260832454 trainer/global_step‚4499 % _timestamp‚1.7843377385028312e+09 _runtime‚7899.088049057 _step‚367 % lr-AdamW/pg1‚0.000432253260832454Å6â¿ÊªÄ % lr-AdamW/pg1‚0.000432253260832454 % lr-AdamW/pg2‚0.000432253260832454 trainer/global_step‚4499 % _timestamp‚1.7843377385028312e+09 _runtime‚7899.088049057 _step‚367ïõê¯S2«- _wandbruntime‚7899  _runtime‚7899ÒÙE¬ùO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚368 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24345788 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1297430554981376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38383431 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚4499 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚354867874274304 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1296864243859715 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38383431 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1292639513882624 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1297431926544384 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1297430554981376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1293198583529472 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38383437 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1297431926544384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1293198557356032 KBDeviceStatsMonitor.on_µ"‘Ϻtrain_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4226100005381 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4224755405059 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12799723 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4233343014912 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38972775 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4231997625344 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1292639488454656 _runtime‚7899.089326772 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1293198557356032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11545802 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38962802 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24345520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12799986 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5457651011584 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1293198557356032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11545802 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38962802 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24345520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12799986 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5457651011584 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1297431926544384 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1297430554981376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1293198583529472 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192ðodp2®- _wandbruntime‚7900  _runtime‚7900%GåÑ '¯ƒN REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38981267 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38391637 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1297710852317184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 % _timestamp‚1.7843377398998964e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4232922660864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚4499 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1293476558086144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12802566 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _step‚369 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4227023471739 _runtime‚7900.485199942 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77362917 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38981267 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1297710852317184 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1297709480747008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38391637 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24350911 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354946550622208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349487675133952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77362917 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38391643 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5458875488256 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1297144419132539 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1293476584259584 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4234268057600 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24350647 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1293476584259584 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77372910 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1293476558086144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38971280 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349487307042816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77372910 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38391643 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1297143049104137 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12802307 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38971280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1292917395660800 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1297710852317184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 % _timestamp‚1.7843377398998964e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4232922660864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚4499 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1293476558086144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12802566 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _step‚369 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4227023471739 _runtime‚7900.485199942 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77362917 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38981267 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1297710852317184 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1297709480747008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38391637 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24350911 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354946550622208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349487675133952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77362917 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38391643 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.all€¯Wn*ocated‚ 5458875488256 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1297144419132539 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1293476584259584 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4234268057600 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24350647 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1293476584259584 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77372910 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1293476558086144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38971280 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349487307042816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77372910 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38391643 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1297143049104137 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12802307 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38971280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1292917395660800 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1306533577363456 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78192264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚353284509110272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _runtime‚7979.887108198 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1311376907417600 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4272337118489 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4270992518167 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 % _timestamp‚1.7843378193018632e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353284141019136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1310804569881623 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11672952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5516981723136 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38793737 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1311378278980608 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1307098620035072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4278313555968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11672947 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39388554 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1311376907417600 trainer/global_step‚4549 _step‚372 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5516945089536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1306533602791424 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24608272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1307098620035072 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4279658945536 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12935320 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4278313555968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24607999 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12935052 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322Z3³#ÃþO VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4270992518167 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 % _timestamp‚1.7843378193018632e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353284141019136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1310804569881623 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11672952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5516981723136 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38793737 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1311378278980608 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1307098620035072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4278313555968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11672947 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39388554 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1311376907417600 trainer/global_step‚4549 _step‚372 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5516945089536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1306533602791424 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24608272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1307098620035072 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4279658945536 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12935320 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4278313555968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24607999 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12935052 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1310805939909913 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1306533577363456 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78192264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚353284509110272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _runtime‚7979.887108198 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1311376907417600 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4272337118489 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216ô á@ŸÄ‚(:” ›«ëÒ«ÚÉ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.958' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.991600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙûžÞÅ‚(:Ó ›«ëÒ²äË$ disk.nvme0c0n1.out‚ 90548.1171875 disk.dm-0.in‚ 23669.59375$ memory_percent‚91.88048839569092 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 352.83984375 disk.rbd1.in‚0 disk.rbd1.out‚ 21.30859375 disk.drbd2165.in‚0 cpu‚25.525916234273627( disk./.usagePercent‚41.9278197816735 disk.sda1.out‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2937.41796875 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.out‚ 90548.1171875 disk.dm-0.out‚ 90548.1171875 disk.nvme0n1.in‚ 23669.59375 network.recv‚ 2.01077e+06 disk.dm-1.out‚ 352.83984375 disk.sda.in‚ 0.47265625) proc.memory.availableMB‚ 1330.30078125$ disk./.usageGB‚624.7548141479492 disk.dm-1.in‚ 0.47265625 disk.sda.out‚ 352.83984375! disk.nvme0c0n1.in‚ 23669.59375* proc.memory.percent‚17.928576469421387 proc.cpu.threads‚17 network.sent‚ 8.736021e+06 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0S9ªË2Æ- _wandbruntime‚7981  _runtime‚7981s¡Ð[ 'ǃN A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1307376594591744 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4280583988224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5518156973056 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 % _timestamp‚1.7843378206964707e+09 _runtime‚7981.281740818 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38801937 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1311084745154447 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1306811484569600 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11675495 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1306811459141632 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39407019 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1311657204753408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38801937 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4279238591488 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1307376620765184 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78208962 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1311655833183232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38801943 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78198969 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12937720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353361627426816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 trainer/global_step‚4549 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39407019 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24613210 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ñ)©'ȈN ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11675495 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1306811459141632 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39407019 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1311657204753408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38801937 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4279238591488 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1307376620765184 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78208962 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1311655833183232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38801943 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78198969 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12937720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353361627426816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 trainer/global_step‚4549 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39407019 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24613210 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1307376594591744 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4280583988224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5518156973056 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 % _timestamp‚1.7843378206964707e+09 _runtime‚7981.281740818 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38801937 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1311084745154447 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1306811484569600 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648õ¶•'p2É- _wandbruntime‚7981  _runtime‚7981KŠ® Ê… 0loss/FRt2/assignment_loss‚0.6974551677703857 2loss/SRqqt1/assignment_loss‚0.5016506567750106 /loss/FRt2/detection_loss‚1.6524554819260773 %loss/total_loss‚8.050879879597474 _step‚374 0loss/FRt1/assignment_loss‚1.5630179160959943 1loss/SRqqt1/detection_loss‚0.6532085045996268 1loss/SRqqt2/detection_loss‚0.7376722667461866 /loss/FBt2/detection_loss‚0.6305274353236765 epoch‚11 3loss/SRqqt2/assignment_loss‚0.49633565315997846 3loss/FBt1/assignment_loss‚0.0010580336425415461 2loss/FBt2/assignment_loss‚0.012562266064635055 /loss/FRt1/detection_loss‚0.5043149735595401 trainer/global_step‚4549 /loss/FBt1/detection_loss‚0.6006215239338215 $ _timestamp‚1.784337820779697e+09 _runtime‚7981.364542396(_ œËŠ /loss/FBt1/detection_loss‚0.6006215239338215 $ _timestamp‚1.784337820779697e+09 _runtime‚7981.364542396 0loss/FRt2/assignment_loss‚0.6974551677703857 2loss/SRqqt1/assignment_loss‚0.5016506567750106 /loss/FRt2/detection_loss‚1.6524554819260773 %loss/total_loss‚8.050879879597474 _step‚374 0loss/FRt1/assignment_loss‚1.5630179160959943 1loss/SRqqt1/detection_loss‚0.6532085045996268 1loss/SRqqt2/detection_loss‚0.7376722667461866 /loss/FBt2/detection_loss‚0.6305274353236765 epoch‚11 3loss/SRqqt2/assignment_loss‚0.49633565315997846 3loss/FBt1/assignment_loss‚0.0010580336425415461 2loss/FBt2/assignment_loss‚0.012562266064635055 /loss/FRt1/detection_loss‚0.5043149735595401 trainer/global_step‚4549öà‡0ßÌ‚(:” ª«ëÒ§Ü¨È gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚332.801' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.56020000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É5FêéÍ‚(:Þ ª«ëÒñ‡äÉ network.recv‚ 2.015289e+06 disk.drbd2165.in‚0! disk.dm-0.out‚90556.990234375 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 21.30859375 disk.sda.in‚ 0.47265625 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚90556.990234375 disk.dm-0.in‚23670.55859375 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 353.37109375 proc.cpu.threads‚17 disk.sda1.in‚0# proc.memory.rssMB‚ 2937.41796875 cpu‚25.526192783919743 disk.sda.out‚ 353.37109375$ memory_percent‚91.88079833984375 disk.sda3.in‚ 0.47265625 disk.rbd0.out‚0 disk.dm-1.in‚ 0.47265625 disk.rbd1.in‚0# proc.memory.availableMB‚1330.25$ disk./.usageGB‚624.7550163269043$ disk.nvme0n1.out‚90556.990234375 disk.dm-1.out‚ 353.37109375) disk./.usagePercent‚41.92783335007281$ disk.nvme0c0n1.in‚23670.55859375" disk.nvme0n1.in‚23670.55859375* proc.memory.percent‚17.928576469421387 network.sent‚ 8.788237e+06“$ªŸÎ‚(:” ¹«ëÒ« ¤Ç gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚249.687' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.937400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09S»ëÏ‚(:à ¹«ëÒýÙ„É network.recv‚ 2.020211e+06$ disk./.usageGB‚624.7558250427246 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625( disk.nvme0c0n1.out‚90567.49755859375 cpu‚25.52649105370181 network.sent‚ 8.840775e+06& disk.nvme0n1.out‚90567.49755859375( proc.memory.availableMB‚ 1330.0078125$ memory_percent‚91.88227653503418 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.47265625 disk.sda.out‚ 353.7109375# disk.dm-0.out‚90567.49755859375* proc.memory.percent‚17.928576469421387 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 23671.390625 proc.cpu.threads‚17 disk.sda3.out‚ 353.7109375 disk.dm-1.out‚ 353.7109375 disk.dm-0.in‚ 23671.390625* disk./.usagePercent‚41.927887623670074 disk.nvme0n1.in‚ 23671.390625 disk.rbd1.out‚ 21.37109375 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2937.41796875Ulš2êЂ(:ß È«ëÒŒÚßÎ disk.drbd2165.in‚0$ disk./.usageGB‚624.7560577392578) disk./.usagePercent‚41.92790324012966 disk.sda.out‚ 354.296875 disk.dm-0.in‚ 23838.3046875( proc.memory.availableMB‚ 1329.4921875 disk.rbd0.out‚0& disk.nvme0n1.out‚90573.95166015625 disk.sda1.in‚0 disk.dm-1.in‚ 0.47265625 proc.cpu.threads‚17 network.sent‚ 8.84782e+06 network.recv‚ 2.023467e+06 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 354.296875 disk.rbd1.out‚ 21.37109375 disk.drbd2165.out‚0# disk.dm-0.out‚90573.95166015625* proc.memory.percent‚17.928576469421387 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 23838.3046875( disk.nvme0c0n1.out‚90573.95166015625$ memory_percent‚91.88542366027832# proc.memory.rssMB‚ 2937.41796875 cpu‚25.526900628240465 disk.sda.in‚ 0.47265625! disk.nvme0n1.in‚ 23838.3046875 disk.sda1.out‚0 disk.sda3.in‚ 0.47265625 disk.sda3.out‚ 354.296875 disk.sda2.in‚0Jt\K”Ñ‚(:‰ È«ëÒôÅùÎ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚296.751' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.3502 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´iscŸÒ‚(:” ׫ëÒÉ…ŽÈ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚330.528' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.10560000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uZÌïÓ‚(:ä ׫ëÒ­‰èÉ) disk./.usagePercent‚41.92790068005432 disk.sda2.out‚0 disk.rbd1.out‚ 21.44921875 disk.sda.out‚ 354.78515625 disk.drbd2165.in‚0 disk.dm-0.in‚23839.26953125 disk.sda1.out‚0 disk.sda2.in‚0" disk.dm-0.out‚90587.4599609375 proc.cpu.threads‚17' disk.nvme0c0n1.out‚90587.4599609375 cpu‚25.52725528501891 network.recv‚ 2.026866e+06$ disk./.usageGB‚624.7560195922852 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚23839.26953125" disk.nvme0n1.in‚23839.26953125 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625* proc.memory.percent‚17.928576469421387 network.sent‚ 8.852896e+06 disk.sda1.in‚0 disk.sda3.out‚ 354.78515625 disk.rbd0.in‚0 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 354.78515625 disk.drbd2165.out‚0$ memory_percent‚91.88241958618164% disk.nvme0n1.out‚90587.4599609375 disk.sda3.in‚ 0.47265625' proc.memory.availableMB‚ 1329.984375# proc.memory.rssMB‚ 2937.41796875Ì“ÈrŸÔ‚(:” æ«ëÒàªÇ gpu.0.gpu‚88.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚264.778' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.955600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€òWAæÕ‚(:Û æ«ëÒ«ü°É$ memory_percent‚91.88411235809326 network.recv‚ 2.030014e+06! disk.nvme0n1.in‚ 23840.1015625 disk.rbd0.out‚0 disk.sda.in‚ 0.47265625 disk.dm-0.in‚ 23840.1015625* disk./.usagePercent‚41.927902216099525 disk.sda1.out‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 23840.1015625# proc.memory.rssMB‚ 2937.41796875% disk.nvme0n1.out‚90595.3173828125 disk.sda2.in‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚90595.3173828125 proc.cpu.threads‚17 network.sent‚ 8.85586e+06 disk.sda3.out‚ 355.09375 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 355.09375" disk.dm-0.out‚90595.3173828125$ disk./.usageGB‚624.7560424804688 disk.drbd2165.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚17.928576469421387 cpu‚25.527578815922116 disk.rbd0.in‚0) proc.memory.availableMB‚ 1329.70703125 disk.sda1.in‚0 disk.rbd1.out‚ 21.44921875 disk.sda.out‚ 355.09375 disk.sda3.in‚ 0.47265625Q}àƒ2Ö- _wandbruntime‚8060  _runtime‚8060ãîSuÈ× _step‚375 ' lr-AdamW/pg1‚0.00042986724932389016 ' lr-AdamW/pg2‚0.00042986724932389016 trainer/global_step‚4599 $ _timestamp‚1.784337899969768e+09 _runtime‚8060.554847679$oXÍØÇ ' lr-AdamW/pg1‚0.00042986724932389016 ' lr-AdamW/pg2‚0.00042986724932389016 trainer/global_step‚4599 $ _timestamp‚1.784337899969768e+09 _runtime‚8060.554847679 _step‚375÷´Ñm82Ù- _wandbruntime‚8060  _runtime‚8060ø¨œªBÚùO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4325974876160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39204037 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79028316 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39814306 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1320427666272256 _step‚376 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1324744895903531 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39824279 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1325324631416832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4325974876160 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1320998630367232 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13068191 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1320998656540672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1325323259853824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39204031 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11800102 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362734463848448 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚357158461403136 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39204037 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚362734057026560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚8060.559861899 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1320427666272256 _step‚376 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1324744895903531 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39824279 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1325324631416832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4325974876160 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1320998630367232 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13068191 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1320998656540672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1325323259853824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39204031 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11800102 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362734463848448 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚357158461403136 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39204037 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚362734057026560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚8060.559861899 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4326899918848 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4325554522112 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24873388 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1325603557189632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79035021 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39212243 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11802640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1325023701147953 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1320705573478400 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39832771 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39822784 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1321276631097344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362812720190464 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5576772379648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79045014 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24873668 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362813124908032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚8061.953448188 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1320705548050432 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39832771 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 trainer/global_step‚4599 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39212237 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79045014 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13071023 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362813124908032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚8061.953448188 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1320705548050432 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39832771 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 trainer/global_step‚4599 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39212237 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79045014 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13071023 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4326899918848 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4325554522112 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24873388 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1325603557189632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79035021 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39212243 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11802640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1325023701147953 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1320705573478400 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39832771 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39822784 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1321276631097344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362812720190464 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5576772379648ùü(Ÿ‰2ß- _wandbruntime‚8062  _runtime‚8062èÑF­àú _runtime‚8062.035830386 /loss/FRt2/assignment_loss‚0.887599046460263 2loss/FBt1/assignment_loss‚0.013502098101868575 /loss/SRqqt1/detection_loss‚0.80041383448216 trainer/global_step‚4599 _step‚378 $loss/total_loss‚9.20571959752898 -loss/FRt1/assignment_loss‚1.6513524528846 2loss/SRqqt2/assignment_loss‚0.5674813652804644 2loss/FBt2/assignment_loss‚0.014100798398510207 /loss/FRt1/detection_loss‚0.5669360184466234 .loss/FBt2/detection_loss‚1.022274442948401 % _timestamp‚1.7843379014510465e+09 2loss/SRqqt1/assignment_loss‚0.5361478136961048 .loss/FRt2/detection_loss‚1.773788764609525 1loss/SRqqt2/detection_loss‚0.7615696485154331 .loss/FBt1/detection_loss‚0.610553313705027 epoch‚11¦Ð+áÿ 2loss/SRqqt2/assignment_loss‚0.5674813652804644 2loss/FBt2/assignment_loss‚0.014100798398510207 /loss/FRt1/detection_loss‚0.5669360184466234 .loss/FBt2/detection_loss‚1.022274442948401 % _timestamp‚1.7843379014510465e+09 2loss/SRqqt1/assignment_loss‚0.5361478136961048 .loss/FRt2/detection_loss‚1.773788764609525 1loss/SRqqt2/detection_loss‚0.7615696485154331 .loss/FBt1/detection_loss‚0.610553313705027 epoch‚11 _runtime‚8062.035830386 /loss/FRt2/assignment_loss‚0.887599046460263 2loss/FBt1/assignment_loss‚0.013502098101868575 /loss/SRqqt1/detection_loss‚0.80041383448216 trainer/global_step‚4599 _step‚378 $loss/total_loss‚9.20571959752898 -loss/FRt1/assignment_loss‚1.6513524528846ú Qý”â‚(:‰ õ«ëÒèØñÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d0[éã‚(:Þ õ«ëÒ¡ÅéÉ' disk.nvme0c0n1.out‚90603.2548828125 disk.sda3.in‚ 0.47265625 disk.drbd2165.in‚0 disk.dm-0.in‚ 23840.1171875! disk.nvme0n1.in‚ 23840.1171875 disk.sda2.in‚0! proc.memory.rssMB‚ 2937.421875 cpu‚25.5279262957392$ disk./.usageGB‚624.7613906860352 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 23840.1171875 disk.dm-1.in‚ 0.47265625 proc.cpu.threads‚17 network.sent‚ 8.860847e+06 disk.sda3.out‚ 355.81640625 network.recv‚ 2.033613e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚ 0.47265625 disk.sda.out‚ 355.81640625" disk.dm-0.out‚90603.2548828125# memory_percent‚91.8846607208252 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚90603.2548828125( proc.memory.availableMB‚ 1329.6171875* proc.memory.percent‚17.928600311279297) disk./.usagePercent‚41.92826113866253 disk.dm-1.out‚ 355.81640625 disk.rbd1.out‚ 21.51171875¤# “ä‚(:ˆ „¬ëÒš¹É gpu.0.gpu‚82.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚330.145' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.029 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™¤Òeôå‚(:é „¬ëÒ¨ÒÑË( disk.nvme0c0n1.out‚90610.83056640625 disk.sda.out‚ 356.16015625* proc.memory.percent‚17.928600311279297* disk./.usagePercent‚41.928408599002246 disk.drbd2165.out‚0 disk.dm-0.in‚23953.87109375 disk.sda1.in‚0 network.recv‚ 2.038243e+06& disk.nvme0n1.out‚90610.83056640625 disk.dm-1.in‚ 0.47265625 disk.dm-1.out‚ 356.16015625) proc.memory.availableMB‚ 1330.08984375 disk.rbd0.in‚0 disk.rbd1.out‚ 21.51171875! proc.memory.rssMB‚ 2937.421875 proc.cpu.threads‚17 network.sent‚ 8.923816e+06 disk.drbd2165.in‚0# disk.dm-0.out‚90610.83056640625 cpu‚25.528417621373777 disk.rbd1.in‚0 disk.sda.in‚ 0.47265625$ memory_percent‚91.88177585601807$ disk./.usageGB‚624.7635879516602" disk.nvme0n1.in‚23953.87109375 disk.sda3.in‚ 0.47265625 disk.sda2.in‚0 disk.sda3.out‚ 356.16015625 disk.rbd0.out‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚23953.87109375 disk.sda1.out‚0J†A)Ÿæ‚(:” “¬ëÒí¸¸Ç gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚285.735' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.147000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»ÏÅïç‚(:ä “¬ëÒÑ’¸É network.sent‚ 8.928827e+06" disk.nvme0n1.in‚23953.87109375 disk.sda1.in‚0 disk.sda3.out‚ 356.56640625 disk.dm-1.out‚ 356.56640625 disk.rbd1.in‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚90619.5224609375! proc.memory.rssMB‚ 2937.421875* proc.memory.percent‚17.928600311279297 proc.cpu.threads‚17 disk.sda.in‚ 0.47265625$ memory_percent‚91.88332557678223$ disk./.usageGB‚624.8273811340332 disk.dm-1.in‚ 0.47265625 cpu‚25.528748766122952) disk./.usagePercent‚41.93268981299735 disk.sda2.out‚0$ disk.nvme0c0n1.in‚23953.87109375" disk.dm-0.out‚90619.5224609375% disk.nvme0n1.out‚90619.5224609375 disk.sda3.in‚ 0.47265625 disk.rbd0.in‚0 disk.dm-0.in‚23953.87109375 network.recv‚ 2.041811e+06 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 21.58984375( proc.memory.availableMB‚ 1329.8359375 disk.sda2.in‚0 disk.sda.out‚ 356.56640625 disk.drbd2165.out‚0I²¬Ÿ”è‚(:‰ ¢¬ëÒ¾ž¸Ç gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚246.203' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.2406 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºZ„sçé‚(:Ü ¢¬ëÒ÷ùÊ" disk.dm-0.out‚90630.6787109375 disk.sda3.out‚ 356.8515625 disk.rbd0.in‚0 disk.dm-1.out‚ 356.8515625" disk.nvme0c0n1.in‚ 23954.703125 disk.rbd1.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2937.421875 proc.cpu.threads‚17 network.sent‚ 8.933807e+06$ disk./.usageGB‚624.8248634338379 disk.rbd0.out‚0 disk.sda2.out‚0$ memory_percent‚91.88644886016846* proc.memory.percent‚17.928600311279297 cpu‚25.529029970107054 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚90630.6787109375) proc.memory.availableMB‚ 1329.32421875 network.recv‚ 2.044897e+06 disk.sda3.in‚ 0.47265625 disk.sda2.in‚0 disk.sda.out‚ 356.8515625 disk.drbd2165.out‚0 disk.sda1.in‚0) disk./.usagePercent‚41.93252084802475 disk.rbd1.out‚ 21.58984375 disk.sda.in‚ 0.47265625 disk.dm-0.in‚ 23954.703125 disk.nvme0n1.in‚ 23954.703125% disk.nvme0n1.out‚90630.6787109375 disk.dm-1.in‚ 0.47265625ˆi¹k”ê‚(:‰ ±¬ëÒº”ŸÇ gpu.0.gpu‚86.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚340.866' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.1732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08¹¨µíë‚(:â ±¬ëÒãé¾Ë disk.rbd1.in‚0 network.recv‚ 2.048573e+06* disk./.usagePercent‚41.932508815670644 disk.dm-0.in‚23955.66796875! disk.dm-0.out‚90705.021484375 disk.sda1.out‚0 disk.sda3.out‚ 357.44140625! proc.memory.rssMB‚ 2937.421875$ disk.nvme0c0n1.in‚23955.66796875 disk.sda1.in‚0 disk.rbd0.in‚0 cpu‚25.52942666058508 disk.dm-1.in‚ 0.47265625 disk.rbd1.out‚ 21.71484375 disk.sda.in‚ 0.47265625$ memory_percent‚91.88368320465088 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda.out‚ 357.44140625 disk.drbd2165.in‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1329.77734375 proc.cpu.threads‚17 network.sent‚ 8.938894e+06& disk.nvme0c0n1.out‚90705.021484375 disk.sda2.in‚0* proc.memory.percent‚17.928600311279297$ disk./.usageGB‚624.8246841430664" disk.nvme0n1.in‚23955.66796875$ disk.nvme0n1.out‚90705.021484375 disk.sda3.in‚ 0.47265625 disk.dm-1.out‚ 357.44140625æGoU2ì- _wandbruntime‚8140  _runtime‚8140­ìqMÇíÁ _runtime‚8140.814878849 _step‚379 & lr-AdamW/pg1‚0.0004286560319798416 & lr-AdamW/pg2‚0.0004286560319798416 trainer/global_step‚4649 % _timestamp‚1.7843379802297728e+09¸-ÐêÌîÆ & lr-AdamW/pg2‚0.0004286560319798416 trainer/global_step‚4649 % _timestamp‚1.7843379802297728e+09 _runtime‚8140.814878849 _step‚379 & lr-AdamW/pg1‚0.0004286560319798416û’0ö2ï- _wandbruntime‚8140  _runtime‚8140é—‘þ'ðøO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1338685221925439 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚380 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5634517485056 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13203261 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚366667703996928 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1334321755181056 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5634554118656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4364811344705 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39614331 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40240058 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40250031 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79854389 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚366667299272192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚ 8140.81982276 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1334898693046272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39614337 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39614337 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39614331 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40240058 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4370945417216 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1334898693046272 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1334898666872832 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1338686591953729 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40250031 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚361033149878272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79864368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4363466744383 trainer/global_step‚4649 % _timestamp‚1.7843379802309036e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1339270983853056 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11927252 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1339269612290048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1334321780609024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1339269612290048 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚361032781787136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11927247 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1339270983853056 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1339269612290048 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚361032781787136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11927247 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1339270983853056 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚ 8140.81982276 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1334898693046272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39614337 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39614337 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39614331 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40240058 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4370945417216 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1334898693046272 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1334898666872832 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1338686591953729 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40250031 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚361033149878272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79864368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4363466744383 trainer/global_step‚4649 % _timestamp‚1.7843379802309036e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1339270983853056 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0üÍÅÝ2ò- _wandbruntime‚8142  _runtime‚8142˜RB²'ó‚N UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1338964027169861 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4364390210629 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25135410 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361110636285952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1334599662387200 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1335176693776384 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79871073 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25135683 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4373215849472 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39622543 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 trainer/global_step‚4649 $ _timestamp‚1.784337981624551e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1335176667602944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4371870452736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1338965397198263 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11929795 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5635728322048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39622543 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40248536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5635691695616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚361110268194816 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79881066 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39622537 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79871073 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39622537 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _step‚381 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4371870452736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1335176693776384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40258523 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504wÓ 'ô‡N T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1338964027169861 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4364390210629 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25135410 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361110636285952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1334599662387200 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1335176693776384 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79871073 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25135683 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4373215849472 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39622543 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 trainer/global_step‚4649 $ _timestamp‚1.784337981624551e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1335176667602944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4371870452736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1338965397198263 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11929795 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5635728322048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39622543 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40248536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5635691695616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚361110268194816 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79881066 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39622537 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79871073 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39622537 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _step‚381 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4371870452736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1335176693776384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40258523 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ý6ίÇ2õ- _wandbruntime‚8142  _runtime‚8142-d} öƒ 1loss/SRqqt2/assignment_loss‚0.550105776726128 /loss/FRt2/detection_loss‚1.5692477862328984 1loss/SRqqt1/detection_loss‚0.7808303743856781 %loss/total_loss‚8.636677854790168 $ _timestamp‚1.784337981708064e+09 _runtime‚8142.292855821 _step‚382 0loss/FRt2/assignment_loss‚0.8012342135573542 3loss/SRqqt1/assignment_loss‚0.49898112795928484 epoch‚11 0loss/FRt1/assignment_loss‚1.6115698886271532 2loss/FBt1/assignment_loss‚0.012983165879904524 /loss/FRt1/detection_loss‚0.5270683980113007 /loss/FBt2/detection_loss‚1.0801117561757565 2loss/FBt2/assignment_loss‚0.004151797077201991 1loss/SRqqt2/detection_loss‚0.6922782052072525 /loss/FBt1/detection_loss‚0.5081153649502554 trainer/global_step‚4649¹»üQ÷ˆ 1loss/SRqqt2/detection_loss‚0.6922782052072525 /loss/FBt1/detection_loss‚0.5081153649502554 trainer/global_step‚4649 1loss/SRqqt2/assignment_loss‚0.550105776726128 /loss/FRt2/detection_loss‚1.5692477862328984 1loss/SRqqt1/detection_loss‚0.7808303743856781 %loss/total_loss‚8.636677854790168 $ _timestamp‚1.784337981708064e+09 _runtime‚8142.292855821 _step‚382 0loss/FRt2/assignment_loss‚0.8012342135573542 3loss/SRqqt1/assignment_loss‚0.49898112795928484 epoch‚11 0loss/FRt1/assignment_loss‚1.6115698886271532 2loss/FBt1/assignment_loss‚0.012983165879904524 /loss/FRt1/detection_loss‚0.5270683980113007 /loss/FBt2/detection_loss‚1.0801117561757565 2loss/FBt2/assignment_loss‚0.004151797077201991þDë~+žø‚(:“ À¬ëÒÚîûÇ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚280.367' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.07340000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ}§âù‚(:× À¬ëÒÁ×É network.sent‚ 8.944005e+06 disk.sda1.out‚0" disk.dm-0.out‚90708.4189453125 proc.cpu.threads‚17$ disk./.usageGB‚624.8252449035645' disk.nvme0c0n1.out‚90708.4189453125 disk.sda3.in‚ 1.41015625 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 24079.3359375 disk.dm-1.in‚ 1.41015625! proc.memory.rssMB‚ 2937.421875% disk.nvme0n1.out‚90708.4189453125 disk.rbd0.in‚0 disk.dm-1.out‚358.6875 disk.sda.out‚358.6875) proc.memory.availableMB‚ 1330.05859375* disk./.usagePercent‚41.932546448778176 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 1.41015625# disk.nvme0c0n1.in‚ 24079.3359375$ memory_percent‚91.88196659088135* proc.memory.percent‚17.928600311279297 cpu‚25.529659775524618 network.recv‚ 2.051972e+06 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 21.71484375 disk.dm-0.in‚ 24079.3359375 disk.sda3.out‚358.6875Ùå,M”ú‚(:‰ ϬëÒœëÀÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s̶Ìíû‚(:â ϬëÒÚµÀÉ$ disk.nvme0n1.out‚90718.126953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 1.41015625 disk.rbd1.out‚ 21.79296875 disk.dm-0.in‚24080.16796875 network.recv‚ 2.056833e+06 disk.dm-1.in‚ 1.41015625 disk.dm-1.out‚ 358.99609375! disk.dm-0.out‚90718.126953125( proc.memory.availableMB‚ 1330.6953125 disk.sda3.out‚ 358.99609375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2165.in‚0$ memory_percent‚91.87808036804199 cpu‚25.530003662326884* disk./.usagePercent‚41.932531344333654 disk.rbd1.in‚0! proc.memory.rssMB‚ 2937.421875 disk.sda.out‚ 358.99609375 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚24080.16796875 network.sent‚ 9.006979e+06$ disk./.usageGB‚624.8250198364258 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda.in‚ 1.41015625& disk.nvme0c0n1.out‚90718.126953125* proc.memory.percent‚17.928600311279297" disk.nvme0n1.in‚24080.16796875 proc.cpu.threads‚17;Ä%”ü‚(:‰ Þ¬ëÒ•ÁÁÇ gpu.0.gpu‚54.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚193.356' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.6712 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v]›êý‚(:ß Þ¬ëÒßæÕË! proc.memory.rssMB‚ 2937.421875 proc.cpu.threads‚17 network.recv‚ 2.060126e+06) disk./.usagePercent‚41.93254337668776 disk.sda.in‚ 1.41015625 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 24081.1328125$ disk./.usageGB‚624.8251991271973 disk.sda.out‚ 359.4453125 disk.sda1.in‚0$ memory_percent‚91.87943935394287* proc.memory.percent‚17.928600311279297( disk.nvme0c0n1.out‚90725.69970703125 disk.rbd1.out‚ 21.79296875 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 24081.1328125 disk.rbd0.in‚0 disk.sda2.out‚0 cpu‚25.530483534367 disk.dm-0.in‚ 24081.1328125 disk.sda2.in‚0 disk.dm-1.in‚ 1.41015625 disk.dm-1.out‚ 359.4453125# disk.dm-0.out‚90725.69970703125 network.sent‚ 9.014001e+06& disk.nvme0n1.out‚90725.69970703125 disk.sda3.out‚ 359.4453125 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 1.41015625) proc.memory.availableMB‚ 1330.47265625’oS¾”þ‚(:‰ í¬ëÒØ—¡Ç gpu.0.gpu‚89.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚275.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑÁÐ<ðÿ‚(:å í¬ëÒð™ùÈ disk.rbd1.in‚0# disk.dm-0.out‚90730.35205078125& disk.nvme0n1.out‚90730.35205078125 disk.sda1.in‚0( proc.memory.availableMB‚ 1330.7109375* disk./.usagePercent‚41.932541072619955 disk.sda.in‚ 1.41015625( disk.nvme0c0n1.out‚90730.35205078125$ memory_percent‚91.87798500061035 network.sent‚ 9.017029e+06 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.drbd2165.out‚0 cpu‚25.530780216205844 disk.sda2.out‚0 disk.dm-1.in‚ 1.41015625 disk.rbd1.out‚ 21.85546875 disk.sda1.out‚0! proc.memory.rssMB‚ 2937.421875* proc.memory.percent‚17.928600311279297 disk.rbd0.in‚0 disk.sda.out‚ 359.92578125 disk.dm-0.in‚ 24081.1328125! disk.nvme0n1.in‚ 24081.1328125$ disk./.usageGB‚624.8251647949219 disk.sda3.in‚ 1.41015625 disk.sda3.out‚ 359.92578125 disk.dm-1.out‚ 359.92578125# disk.nvme0c0n1.in‚ 24081.1328125 proc.cpu.threads‚17 network.recv‚ 2.063906e+06­ÖCŸ€‚(:” ü¬ëÒêžÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚291.194' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.238800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²ìMÝ‚(:Ò ü¬ëÒ’ôÑÉ disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1330.22265625 proc.cpu.threads‚17# disk./.usageGB‚624.825366973877 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 24181.8125 disk.sda3.in‚ 1.41015625 disk.sda2.in‚0) disk./.usagePercent‚41.93255464101927 disk.dm-1.out‚ 360.21484375 disk.sda3.out‚ 360.21484375 disk.rbd1.out‚ 21.85546875 disk.sda.out‚ 360.21484375 disk.dm-0.in‚ 24181.8125 disk.dm-0.out‚90738.22265625! proc.memory.rssMB‚ 2937.421875 disk.dm-1.in‚ 1.41015625 disk.sda.in‚ 1.41015625 disk.nvme0c0n1.in‚ 24181.8125% disk.nvme0c0n1.out‚90738.22265625* proc.memory.percent‚17.928600311279297 network.sent‚ 9.021936e+06# disk.nvme0n1.out‚90738.22265625 cpu‚25.531023370085574 network.recv‚ 2.067128e+06 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0$ memory_percent‚91.88096523284912ÐZ-Ÿ‚‚(:” ‹­ëÒ‹Œ½É gpu.0.gpu‚86.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚127.824' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.564799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0룠m胂(:Ý ‹­ëÒêÚ®Ë" disk.dm-0.out‚90752.3017578125 disk.sda1.out‚0 disk.sda3.in‚ 1.41015625 network.sent‚ 9.028958e+06$ disk./.usageGB‚624.8291282653809 disk.sda.in‚ 1.41015625! proc.memory.rssMB‚ 2937.421875* proc.memory.percent‚17.928600311279297 disk.sda2.out‚0 disk.dm-1.in‚ 1.41015625 disk.dm-1.out‚ 360.76953125 disk.dm-0.in‚ 24197.28125 disk.nvme0n1.in‚ 24197.28125 disk.rbd0.in‚0$ memory_percent‚91.87796115875244 network.recv‚ 2.070514e+06 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚90752.3017578125) proc.memory.availableMB‚ 1330.71484375 disk.rbd0.out‚0* disk./.usagePercent‚41.932807064448035 disk.sda2.in‚0 disk.rbd1.out‚ 21.93359375 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 24197.28125 disk.sda1.in‚0 cpu‚25.531471209512507 proc.cpu.threads‚17% disk.nvme0n1.out‚90752.3017578125 disk.rbd1.in‚0 disk.sda.out‚ 360.76953125 disk.sda3.out‚ 360.76953125™`Þ2„- _wandbruntime‚8221  _runtime‚8221·²É…à _step‚383 ' lr-AdamW/pg1‚0.00042743277704220136 ' lr-AdamW/pg2‚0.00042743277704220136 trainer/global_step‚4699 % _timestamp‚1.7843380608799143e+09 _runtime‚8221.466859119°CaÎ†È _step‚383 ' lr-AdamW/pg1‚0.00042743277704220136 ' lr-AdamW/pg2‚0.00042743277704220136 trainer/global_step‚4699 % _timestamp‚1.7843380608799143e+09 _runtime‚8221.466859119ÿ혔2‡- _wandbruntime‚8221  _runtime‚8221Œ]¯®ý'ˆ÷O ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1348798703378432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚ 8221.46884086 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40665810 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4409703857491 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40665810 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1348798729551872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4418606737408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5693718726144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚4699 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40675783 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1348215844089856 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25390777 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚370600820897280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4418606737408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1353217336289280 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1353215964726272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚384 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12054397 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13336629 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1352625547947347 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1348798703378432 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1348215869517824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80690441 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40024631 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1353217336289280 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1353215964726272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4417261347840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12054402 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1352626917975637 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5693753262592 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4417261347840'-çÏ(‰üO O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚ 8221.46884086 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40665810 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4409703857491 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40665810 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1348798729551872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4418606737408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5693718726144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚4699 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40675783 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1348215844089856 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25390777 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚370600820897280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4418606737408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1353217336289280 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1353215964726272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚384 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12054397 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13336629 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1352625547947347 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1348798703378432 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1348215869517824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80690441 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40024631 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1353217336289280 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1353215964726272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4417261347840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12054402 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1352626917975637 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5693753262592 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4417261347840 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1348798703378432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872€ÿtÜ2Š- _wandbruntime‚8222  _runtime‚8222òòb+ '‹ƒN C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80707125 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40674288 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40032837 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1352905723220171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80717118 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80707125 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1349076704108544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40674288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12056940 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4419531780096 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40032837 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1349076730281984 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚4699 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13338947 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4419531780096 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40032843 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1349076730281984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843380622859685e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12056945 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚370679902567936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364984588578816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364984956669952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4418186383360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40032843 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4418186383360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80717118 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1349076704108544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5694945897984 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1348493725868032 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1353494890491904 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25395887 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13339190 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640•ôy÷ŒˆN ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5694911368704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40674288 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40032837 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1352905723220171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80717118 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80707125 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1349076704108544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40674288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12056940 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4419531780096 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40032837 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1349076730281984 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚4699 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13338947 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4419531780096 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40032843 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1349076730281984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843380622859685e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12056945 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚370679902567936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364984588578816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364984956669952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4418186383360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40032843 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4418186383360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80717118 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1349076704108544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5694945897984 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1348493725868032 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1353494890491904 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25395887 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13339190 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80707125 FÆ2- _wandbruntime‚8222  _runtime‚8222ªÏ“ Ž /loss/FRt2/assignment_loss‚0.903195765465498 /loss/FBt2/detection_loss‚0.7478291522208098 trainer/global_step‚4699 1loss/SRqqt2/detection_loss‚0.7808395170955725 /loss/FBt1/detection_loss‚0.5016881845731405 epoch‚11 0loss/FRt1/assignment_loss‚1.6896691806115793 2loss/SRqqt1/assignment_loss‚0.5293406605905506 2loss/SRqqt2/assignment_loss‚0.5935804875735717 0loss/FBt1/assignment_loss‚0.0124753956195385 3loss/FBt2/assignment_loss‚0.0034378697375059514 /loss/FRt2/detection_loss‚1.7303338512498885 _runtime‚ 8222.95220916 _step‚386 /loss/FRt1/detection_loss‚0.4873164452194109 1loss/SRqqt1/detection_loss‚0.7067793809909919 %loss/total_loss‚8.686485890948058 % _timestamp‚1.7843380623674338e+09›Á † 1loss/SRqqt2/detection_loss‚0.7808395170955725 /loss/FBt1/detection_loss‚0.5016881845731405 epoch‚11 0loss/FRt1/assignment_loss‚1.6896691806115793 2loss/SRqqt1/assignment_loss‚0.5293406605905506 2loss/SRqqt2/assignment_loss‚0.5935804875735717 0loss/FBt1/assignment_loss‚0.0124753956195385 3loss/FBt2/assignment_loss‚0.0034378697375059514 /loss/FRt2/detection_loss‚1.7303338512498885 _runtime‚ 8222.95220916 _step‚386 /loss/FRt1/detection_loss‚0.4873164452194109 1loss/SRqqt1/detection_loss‚0.7067793809909919 %loss/total_loss‚8.686485890948058 % _timestamp‚1.7843380623674338e+09 /loss/FRt2/assignment_loss‚0.903195765465498 /loss/FBt2/detection_loss‚0.7478291522208098 trainer/global_step‚4699‚ò™ë¸”‚(:‰ š­ëÒÚÕ€È gpu.0.gpu‚82.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚168.264' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.6528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z=¹ËÑ‘‚(:Æ š­ëÒ‡ÍßÉ disk.sda3.in‚ 1.41015625 disk.sda2.out‚0 disk.dm-1.out‚361.125 disk.rbd1.out‚ 21.93359375# memory_percent‚91.8813705444336 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 24197.28125$ disk./.usageGB‚624.8713722229004$ disk.nvme0n1.out‚90755.732421875 disk.sda3.out‚361.125 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.out‚361.125! proc.memory.rssMB‚ 2937.421875 network.sent‚ 9.031981e+06 disk.dm-1.in‚ 1.41015625& proc.memory.availableMB‚ 1330.15625* proc.memory.percent‚17.928600311279297 proc.cpu.threads‚17 disk.nvme0n1.in‚ 24197.28125 disk.sda.in‚ 1.41015625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 cpu‚25.531636387332682) disk./.usagePercent‚41.93564209188221! disk.nvme0c0n1.in‚ 24197.28125& disk.nvme0c0n1.out‚90755.732421875! disk.dm-0.out‚90755.732421875 network.recv‚ 2.073812e+06 disk.rbd0.in‚0itúQž’‚(:“ ©­ëÒìÐßÈ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03eXñ“‚(:æ ©­ëÒù‰ÚÉ* proc.memory.percent‚17.928600311279297# disk./.usageGB‚624.871036529541 disk.sda2.out‚0 disk.drbd2165.in‚0" disk.nvme0n1.in‚24198.11328125$ memory_percent‚91.88249111175537 network.sent‚ 9.094921e+06$ disk.nvme0c0n1.in‚24198.11328125) proc.memory.availableMB‚ 1329.97265625 disk.dm-1.out‚ 361.38671875 disk.sda.out‚ 361.38671875 disk.sda1.in‚0 disk.sda3.in‚ 1.41015625 proc.cpu.threads‚17 network.recv‚ 2.078683e+06 disk.rbd1.in‚0 disk.dm-0.in‚24198.11328125 disk.sda3.out‚ 361.38671875 disk.dm-1.in‚ 1.41015625 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚90767.12353515625& disk.nvme0n1.out‚90767.12353515625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.531881411549655 disk.rbd1.out‚ 21.99609375 disk.sda.in‚ 1.41015625# disk.dm-0.out‚90767.12353515625! proc.memory.rssMB‚ 2937.421875( disk./.usagePercent‚41.9356195632192 disk.sda2.in‚0ñƒ“”‚(:ˆ ¸­ëÒ¾ÆùÇ gpu.0.gpu‚19.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐÐs³ä•‚(:Ù ¸­ëÒ“ŠË disk.sda3.in‚ 1.41015625 disk.rbd1.out‚ 21.99609375 disk.sda.in‚ 1.41015625 disk.sda.out‚ 361.7890625 network.sent‚ 9.10183e+06) disk./.usagePercent‚41.93561572310618% disk.nvme0n1.out‚90775.7158203125 disk.dm-1.in‚ 1.41015625( proc.memory.availableMB‚ 1330.0859375 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 24199.078125 disk.dm-0.in‚ 24199.078125 cpu‚25.532327254884617$ memory_percent‚91.88179969787598 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0# disk./.usageGB‚624.870979309082 disk.sda3.out‚ 361.7890625 disk.rbd1.in‚0! proc.memory.rssMB‚ 2937.421875 disk.rbd0.in‚0 disk.dm-1.out‚ 361.7890625 disk.drbd2165.in‚0* proc.memory.percent‚17.928600311279297 proc.cpu.threads‚17 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚90775.7158203125" disk.dm-0.out‚90775.7158203125 network.recv‚ 2.082055e+06 disk.nvme0n1.in‚ 24199.078125 disk.sda1.out‚0’®ëj”–‚(:‰ Ç­ëÒ‚©ýÇ gpu.0.gpu‚93.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚244.554' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.9108 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WÅ`ì—‚(:á Ç­ëÒñÕÔÉ( disk./.usagePercent‚41.9356577083418& disk.nvme0n1.out‚90825.39501953125 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 1.41015625 disk.rbd0.in‚0 disk.sda.in‚ 1.41015625$ memory_percent‚91.88253879547119) proc.memory.availableMB‚ 1329.96484375 disk.sda2.in‚0! proc.memory.rssMB‚ 2937.421875 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 22.07421875( disk.nvme0c0n1.out‚90825.39501953125# disk.dm-0.out‚90825.39501953125 network.sent‚ 9.104858e+06 disk.sda3.out‚ 362.2734375# disk.nvme0c0n1.in‚ 24294.5390625* proc.memory.percent‚17.928600311279297 cpu‚25.532639478952348 proc.cpu.threads‚17$ disk./.usageGB‚624.8716049194336 disk.rbd0.out‚0 disk.dm-1.in‚ 1.41015625 disk.sda.out‚ 362.2734375 disk.dm-0.in‚ 24294.5390625! disk.nvme0n1.in‚ 24294.5390625 disk.dm-1.out‚ 362.2734375 network.recv‚ 2.085557e+06 disk.sda1.out‚0 disk.drbd2165.in‚0„5Ó%“˜‚(:ˆ Ö­ëÒö¥ßÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚262.895' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.579 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jÛ DꙂ(:ß Ö­ëÒÆËÛÉ disk.rbd0.in‚0# disk.dm-0.out‚90833.18212890625 proc.cpu.threads‚17* disk./.usagePercent‚41.935660012409606 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚24295.37109375 cpu‚25.532520193609024 network.recv‚ 2.088084e+06 disk.rbd1.out‚ 22.07421875$ disk.nvme0c0n1.in‚24295.37109375$ memory_percent‚91.88106060028076" disk.nvme0n1.in‚24295.37109375& disk.nvme0n1.out‚90833.18212890625 disk.sda3.in‚ 1.41015625 disk.sda2.in‚0 disk.dm-1.in‚ 1.41015625# disk./.usageGB‚624.871639251709 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda.out‚ 362.53125( disk.nvme0c0n1.out‚90833.18212890625 disk.dm-1.out‚ 362.53125 network.sent‚ 9.108905e+06 disk.rbd0.out‚0) proc.memory.availableMB‚ 1330.20703125 disk.sda1.out‚0 disk.sda.in‚ 1.41015625* proc.memory.percent‚17.928600311279297 disk.sda3.out‚ 362.53125 disk.rbd1.in‚0! proc.memory.rssMB‚ 2937.421875Rs²2š- _wandbruntime‚8301  _runtime‚8301_›zÏÅ›¿ _runtime‚8301.877131081 _step‚387 % lr-AdamW/pg1‚0.000426197563078348 % lr-AdamW/pg2‚0.000426197563078348 trainer/global_step‚4749 % _timestamp‚1.7843381412919602e+09àÌ‹ÊœÄ % lr-AdamW/pg1‚0.000426197563078348 % lr-AdamW/pg2‚0.000426197563078348 trainer/global_step‚4749 % _timestamp‚1.7843381412919602e+09 _runtime‚8301.877131081 _step‚387ƒá“¨2- _wandbruntime‚8301  _runtime‚8301ͽ‘Xþ'žøO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4463577278464 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5753226106368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5753260642816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1367162317162496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81536472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1362698739884032 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41091562 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41101535 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 _step‚388 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1366567243997545 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1367163688725504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12181547 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1362109958426624 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1362698766057472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12181552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13471264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚374535051289088 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚368781422555136 trainer/global_step‚4749 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40434931 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 $ _timestamp‚1.784338141293265e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1362698739884032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40434931 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4463577278464 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25652816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1367163688725504 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81526493 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4455940970599 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1362698766057472 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1367162317162496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4464922668032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13471018 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _runtime‚8301.878473535 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚374534648661504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4464922668032 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚374535051289088 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚368781422555136 trainer/global_step‚4749 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40434931 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 $ _timestamp‚1.784338141293265e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1362698739884032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40434931 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4463577278464 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25652816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1367163688725504 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81526493 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4455940970599 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMon·ÁîDitor.on_train_batch_start/segment.small_pool.peak‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1362698766057472 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1367162317162496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4464922668032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13471018 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _runtime‚8301.878473535 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚374534648661504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4464922668032 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1367162317162496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81536472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1362698739884032 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41091562 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41101535 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 _step‚388 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1366567243997545 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1367163688725504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12181547 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1362109958426624 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1362698766057472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12181552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13471264„$€2 - _wandbruntime‚8303  _runtime‚8303¾í1 '¡ƒN N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4465847710720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13473961 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5754449239552 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4749 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81553170 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40443137 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚374613726293504 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚374613323673088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1367442614498304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25657808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12184090 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13473718 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚368859277053952 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81553170 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41110027 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4465847710720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40443137 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1362976740614144 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1367441242928128 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41100040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4456864436845 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚368858908962816 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1366846049242079 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚389 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1367442614498304 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1362387814776832 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41110027 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 % _timestamp‚1.7843381426879194e+09rTù"'¢ˆN KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1362976740614144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1362387840204800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4464502313984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1367442614498304 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1362387814776832 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41110027 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 % _timestamp‚1.7843381426879194e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4465847710720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13473961 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5754449239552 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4749 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81553170 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40443137 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚374613726293504 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚374613323673088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1367442614498304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25657808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12184090 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13473718 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚368859277053952 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81553170 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41110027 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4465847710720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40443137 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1362976740614144 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1367441242928128 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41100040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4456864436845 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚368858908962816 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1366846049242079 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚389 QDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41527287 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5812796845568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1381108669598720 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚372656111030272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41527287 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41517314 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1381108669598720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4503522684029 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1376598802563072 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5812762309120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82362545 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1376598802563072 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13606770 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40845231 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1381110041161728 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4509893209088 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚372655742939136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚392 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41517314 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4511238598656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _runtime‚8382.027299036 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1376004047335424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4511238598656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12308697 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13606515 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1376004021907456 $ _timestamp‚1.784338221442025e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41527287 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5812796845568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1381108669598720 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚372656111030272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41527287 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41517314 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1381108669598720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4503522684029 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1376598802563072 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5812762309120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82362545 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1376598802563072 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13606770 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40845231 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1381110041161728 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4509893209088 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚372655742939136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚392 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41517314 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4511238598656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _runtime‚8382.027299036 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1376004047335424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4511238598656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008ˆ`2¶- _wandbruntime‚8383  _runtime‚8383‡¶ »"·ƒN \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378547582240256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12311240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 % _timestamp‚1.7843382228393266e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 _runtime‚8383.424567814 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1381388966934528 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4510818244608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _step‚393 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40853443 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 trainer/global_step‚4799 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12311245 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4503101549953 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41535779 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25920682 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1376876777119744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41535779 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚372733229346816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5813950273024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25920429 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4510818244608 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40853437 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13609437 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1376876777119744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40853443 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4504446150387 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82379229 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚372733597437952 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41525792 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13762819EZ`qN29113600 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82379229 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4512163641344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40853437 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4512163641344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82389222 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82389222 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41525792ÃRµ@'¸ˆN FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25920429 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4510818244608 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40853437 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13609437 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1376876777119744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40853443 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4504446150387 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82379229 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚372733597437952 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41525792 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1376281929113600 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82379229 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4512163641344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40853437 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4512163641344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82389222 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82389222 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41525792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378547582240256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12311240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 % _timestamp‚1.7843382228393266e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 _runtime‚8383.424567814 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1381388966934528 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4510818244608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _step‚393 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40853443 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 trainer/global_step‚4799 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12311245 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4503101549953 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41535779 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25920682 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1376876777119744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41535779 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚372733229346816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5813950273024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995‰ Û2¹- _wandbruntime‚8383  _runtime‚83835j¦ ºƒ /loss/FRt2/detection_loss‚1.7959411420090257 % _timestamp‚1.7843382229213467e+09 /loss/FRt1/assignment_loss‚1.659438560168971 2loss/SRqqt1/assignment_loss‚0.5298601726034562 /loss/FRt1/detection_loss‚0.5203449021900309 1loss/SRqqt1/detection_loss‚0.7768684465881377 trainer/global_step‚4799 _step‚394 2loss/SRqqt2/assignment_loss‚0.5381274291290147 2loss/FBt1/assignment_loss‚0.013310584010419856 1loss/FBt2/assignment_loss‚0.01596861493245082 0loss/FBt1/detection_loss‚0.47823077523608465 /loss/FBt2/detection_loss‚0.9207174305404936 %loss/total_loss‚8.737855839660906 1loss/SRqqt2/detection_loss‚0.7093502854058342 epoch‚11 _runtime‚8383.506124891 0loss/FRt2/assignment_loss‚0.7796974968469866җûˆ %loss/total_loss‚8.737855839660906 1loss/SRqqt2/detection_loss‚0.7093502854058342 epoch‚11 _runtime‚8383.506124891 0loss/FRt2/assignment_loss‚0.7796974968469866 /loss/FRt2/detection_loss‚1.7959411420090257 % _timestamp‚1.7843382229213467e+09 /loss/FRt1/assignment_loss‚1.659438560168971 2loss/SRqqt1/assignment_loss‚0.5298601726034562 /loss/FRt1/detection_loss‚0.5203449021900309 1loss/SRqqt1/detection_loss‚0.7768684465881377 trainer/global_step‚4799 _step‚394 2loss/SRqqt2/assignment_loss‚0.5381274291290147 2loss/FBt1/assignment_loss‚0.013310584010419856 1loss/FBt2/assignment_loss‚0.01596861493245082 0loss/FBt1/detection_loss‚0.47823077523608465 /loss/FBt2/detection_loss‚0.9207174305404936Šõ5.*”¼‚(:‰ °®ëÒá›÷É gpu.0.gpu‚58.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_šRò½‚(:ç °®ëÒñ½“Ì disk.drbd2165.out‚0# disk.dm-0.out‚90973.04541015625$ memory_percent‚91.88320636749268 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0) disk./.usagePercent‚41.94048754648277 disk.sda3.out‚ 368.84765625 proc.cpu.threads‚17 network.sent‚ 9.209845e+06 network.recv‚ 2.109953e+06 disk.sda2.out‚0" disk.nvme0n1.in‚24390.80859375 cpu‚25.53495103277977$ disk./.usageGB‚624.9435729980469 disk.sda3.in‚ 1.55078125 disk.dm-1.in‚ 1.55078125 disk.rbd1.in‚0 disk.sda.in‚ 1.55078125* proc.memory.percent‚17.928600311279297 disk.dm-1.out‚ 368.84765625 disk.rbd1.out‚ 22.27734375$ disk.nvme0c0n1.in‚24390.80859375 disk.dm-0.in‚24390.80859375& disk.nvme0n1.out‚90973.04541015625! proc.memory.rssMB‚ 2937.421875 disk.sda.out‚ 368.84765625( disk.nvme0c0n1.out‚90973.04541015625 disk.sda1.in‚0) proc.memory.availableMB‚ 1329.85546875 disk.drbd2165.in‚0ßC6Δ¾‚(:‰ ¿®ëÒÆñ£É gpu.0.gpu‚98.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚256.693' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.3386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pæ¿‚(:Û ¿®ëÒÁ‘®Ë disk.dm-1.in‚ 1.55078125 disk.dm-1.out‚ 369.453125 disk.rbd1.in‚0$ disk.nvme0n1.out‚90978.689453125 disk.sda1.out‚0! disk.dm-0.out‚90978.689453125& disk.nvme0c0n1.out‚90978.689453125) proc.memory.availableMB‚ 1330.39453125# proc.memory.rssMB‚ 2937.46484375 network.sent‚ 9.259761e+06 network.recv‚ 2.11443e+06 disk.sda1.in‚0 disk.sda.in‚ 1.55078125 disk.sda.out‚ 369.453125) proc.memory.percent‚17.92886257171631 cpu‚25.539552400449562 disk.rbd0.out‚0# disk./.usageGB‚624.943660736084 disk.sda2.in‚0$ disk.nvme0c0n1.in‚24523.92578125 disk.dm-0.in‚24523.92578125" disk.nvme0n1.in‚24523.92578125 disk.sda2.out‚0 disk.rbd1.out‚ 22.35546875 disk.drbd2165.in‚0$ memory_percent‚91.87991619110107 proc.cpu.threads‚17) disk./.usagePercent‚41.94049343465606 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 1.55078125 disk.sda3.out‚ 369.4531253Zѧ2À- _wandbruntime‚8411  _runtime‚8411ì¢åŠy=Áóz 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5832630395889282 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5934869647026062 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4560772478580475 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6851974725723267 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06256324797868729 jet/accuracy_5_of_5‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981371760368347 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5128214359283447 *particle/sensitivity‚0.625646710395813 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6451994776725769 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5979006886482239 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30208513140678406 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48764243721961975 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15732231736183167 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5666739344596863 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.31136414408683777 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9243220090866089 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03850730136036873 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30373769998550415 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2918244004249573 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 6validation_average_jet_accuracy‚0.3593909740447998 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.840684175491333 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6670646667480469 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 4 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9909628033638 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13750000298023224 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987982511520386 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8260393738746643 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3478839099407196 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07137332111597061 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5128214359283447 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.574097216129303 particle/accuracy_3_of_4‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 particle/accuracy_3_of_5‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34735462069511414 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.12078309059143066 *jet/accuracy_2_of_3‚0.5590430498123169 particle/accuracy_1_of_2‚1 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2910662591457367 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9907849431037903 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9949334263801575 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6953480839729309 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 (particle/accuracy‚0.8505511283874512 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6695753931999207 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3385416567325592 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28548845648765564 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14839433133602142 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5784554481506348 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 'particle/f_score‚0.6426020860671997 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5279050469398499 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6207478642463684 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2889957129955292 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +jet/accuracy_3_of_4‚0.08955217897891998 % _timestamp‚1.7843382505787897e+09 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5925341844558716 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9337714910507202 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5654618740081787 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7464096546173096 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2820127010345459 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5314591526985168 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6912396550178528 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965701103210449 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6061345338821411 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5462499856948853 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 epoch‚11 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.29924339056015015 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8733115196228027 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3133941888809204 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8969331383705139 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30268871784210205 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.819255530834198 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9961041808128357 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6444793343544006 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.30000337958335876 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.49512648582458496 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22920304536819458 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6353399753570557 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6082136034965515 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12883888185024261 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 /particle/accuracy_4_of_4‚0.9424124956130981 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4861351251602173 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22022879123687744 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5466359257698059 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.37772300839424133 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9958409070968628 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5186048746109009 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_1_of_5‚1 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.322260320186615 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8714091777801514 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.30329301953315735 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3065534830093384 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14713962376117706 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8417044878005981 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7091240882873535 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9120646119117737 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5660456418991089 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1502036303281784 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.15303747355937958 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6513684391975403 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_3_of_3‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7569964528083801 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 particle/accuracy_2_of_5‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5693439841270447 *jet/accuracy_1_of_4‚0.9632287621498108 particle/accuracy_4_of_5‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5156258940696716 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.056383922696113586 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3028353452682495 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6557610630989075 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 *jet/accuracy_2_of_2‚0.1452162116765976 *jet/accuracy_1_of_3‚0.9373736381530762 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚8.290391171143343 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.32004714012145996 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5411303639411926 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28644195199012756 trainer/global_step‚4799 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9967021942138672 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6190890073776245 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5784922242164612 *jet/accuracy_1_of_2‚0.7792072296142578 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5892676115036011 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.29223892092704773 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6633914113044739 jet/accuracy_4_of_5‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6376209259033203 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981371760368347 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15498384833335876 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10916297882795334 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7569964528083801 particle/accuracy_2_of_2‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 jet/accuracy_1_of_5‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458134651184 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4799869954586029 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12883888185024261 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7476341128349304 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9921452403068542 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9819256663322449 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3819197714328766 jet/accuracy_3_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28431954979896545 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.29223892092704773 *jet/accuracy_2_of_4‚0.7901894450187683 _step‚395 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9986832737922668 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8303477764129639 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.3090919554233551 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982648491859436 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3380310535430908 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.28917229175567627 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04889029636979103 _runtime‚8411.164600267 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 particle/accuracy_1_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_1‚0.3142551779747009 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9907391667366028 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.298797607421875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688Oþ Â\Âøz 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.574097216129303 particle/accuracy_3_of_4‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 particle/accuracy_3_of_5‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34735462069511414 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.12078309059143066 *jet/accuracy_2_of_3‚0.5590430498123169 particle/accuracy_1_of_2‚1 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2910662591457367 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9907849431037903 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9949334263801575 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6953480839729309 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 (particle/accuracy‚0.8505511283874512 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6695753931999207 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3385416567325592 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28548845648765564 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14839433133602142 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5784554481506348 & Purity/2FRt1SxËp"8Rqqt*FBt/FBt_purity‚1 'particle/f_score‚0.6426020860671997 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5279050469398499 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6207478642463684 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2889957129955292 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +jet/accuracy_3_of_4‚0.08955217897891998 % _timestamp‚1.7843382505787897e+09 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5925341844558716 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9337714910507202 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5654618740081787 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7464096546173096 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2820127010345459 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5314591526985168 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6912396550178528 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965701103210449 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6061345338821411 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5462499856948853 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 epoch‚11 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.29924339056015015 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8733115196228027 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3133941888809204 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8969331383705139 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30268871784210205 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.819255530834198 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9961041808128357 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6444793343544006 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.30000337958335876 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.49512648582458496 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22920304536819458 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6353399753570557 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6082136034965515 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12883888185024261 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 /particle/accuracy_4_of_4‚0.9424124956130981 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4861351251602173 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22022879123687744 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5466359257698059 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.37772300839424133 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9958409070968628 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5186048746109009 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_1_of_5‚1 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.322260320186615 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8714091777801514 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.30329301953315735 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3065534830093384 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14713962376117706 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8417044878005981 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7091240882873535 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9120646119117737 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5660456418991089 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1502036303281784 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.15303747355937958 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6513684391975403 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_3_of_3‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7569964528083801 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 particle/accuracy_2_of_5‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5693439841270447 *jet/accuracy_1_of_4‚0.9632287621498108 particle/accuracy_4_of_5‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5156258940696716 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.056383922696113586 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3028353452682495 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6557610630989075 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 *jet/accuracy_2_of_2‚0.1452162116765976 *jet/accuracy_1_of_3‚0.9373736381530762 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚8.290391171143343 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.32004714012145996 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5411303639411926 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28644195199012756 trainer/global_step‚4799 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9967021942138672 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6190890073776245 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5784922242164612 *jet/accuracy_1_of_2‚0.7792072296142578 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5892676115036011 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.29223892092704773 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6633914113044739 jet/accuracy_4_of_5‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6376209259033203 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981371760368347 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15498384833335876 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10916297882795334 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7569964528083801 particle/accuracy_2_of_2‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 jet/accuracy_1_of_5‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458134651184 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4799869954586029 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12883888185024261 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7476341128349304 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9921452403068542 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9819256663322449 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3819197714328766 jet/accuracy_3_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28431954979896545 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.29223892092704773 *jet/accuracy_2_of_4‚0.7901894450187683 _step‚395 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9986832737922668 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8303477764129639 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.3090919554233551 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982648491859436 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3380310535430908 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.28917229175567627 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04889029636979103 _runtime‚8411.164600267 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 particle/accuracy_1_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_1‚0.3142551779747009 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9907391667366028 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.298797607421875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5832630395889282 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5934869647026062 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4560772478580475 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6851974725723267 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06256324797868729 jet/accuracy_5_of_5‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981371760368347 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5128214359283447 *particle/sensitivity‚0.625646710395813 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6451994776725769 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5979006886482239 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30208513140678406 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48764243721961975 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15732231736183167 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5666739344596863 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.31136414408683777 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9243220090866089 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03850730136036873 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30373769998550415 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2918244004249573 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 6validation_average_jet_accuracy‚0.3593909740447998 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.840684175491333 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6670646667480469 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 4 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9909628033638 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13750000298023224 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987982511520386 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8260393738746643 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3478839099407196 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07137332111597061 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5128214359283447‹M‹þ’Â(:‡ ήëÒ£ÜË gpu.0.gpu‚28.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚30.0 gpu.0.powerWatts‚91.796' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3592 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è;Á­éÄ‚(:Þ ήëÒàÂåÌ disk.nvme0n1.in‚ 24524.890625 disk.sda3.out‚ 369.828125) proc.memory.availableMB‚ 1968.72265625* proc.memory.percent‚23.760485649108887 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 1.55078125 network.sent‚ 9.277983e+06 disk.rbd1.out‚ 22.35546875 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 24524.890625 disk.sda1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17# disk.dm-0.out‚90987.70166015625$ memory_percent‚87.98387050628662# proc.memory.rssMB‚ 3892.91796875) disk./.usagePercent‚41.94079680358414 disk.dm-0.in‚ 24524.890625 disk.sda2.out‚0 disk.sda.out‚ 369.828125& disk.nvme0n1.out‚90987.70166015625 disk.rbd0.out‚0 network.recv‚ 2.117919e+06 disk.sda1.out‚0 disk.sda3.in‚ 1.55078125 disk.dm-1.out‚ 369.828125 cpu‚25.542820118877636$ disk./.usageGB‚624.9481811523438( disk.nvme0c0n1.out‚90987.70166015625 disk.rbd1.in‚0 disk.sda.in‚ 1.55078125œ·±”Å‚(:‰ Ý®ëÒ‘ÚÁÇ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚238.122' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.6244 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0а;êçÆ‚(:Ü Ý®ëÒÕÓ¸É$ disk.nvme0c0n1.in‚24526.36328125" disk.nvme0n1.out‚ 92015.6640625 disk.sda1.in‚0$ disk.nvme0c0n1.out‚ 92015.6640625 disk.sda3.out‚ 370.16015625 network.sent‚ 9.322333e+06 disk.sda3.in‚ 1.55078125 proc.cpu.threads‚17# disk./.usageGB‚625.322883605957 disk.dm-0.out‚ 92015.6640625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.959904670715332 cpu‚25.54146431856905 disk.rbd1.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0$ memory_percent‚82.18472003936768 network.recv‚ 2.122343e+06 disk.dm-1.in‚ 1.55078125 disk.dm-1.out‚ 370.16015625 disk.rbd1.out‚ 22.41796875 disk.drbd2165.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2942.55078125) disk./.usagePercent‚41.96594339964243 disk.sda.in‚ 1.55078125 disk.sda.out‚ 370.16015625 disk.dm-0.in‚24526.36328125" disk.nvme0n1.in‚24526.36328125) proc.memory.availableMB‚ 2918.85546875fÛô¢”Ç‚(:‰ ì®ëÒšš²Ç gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚273.606' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+pÔâÈ‚(:× ì®ëÒû‰¨É( disk.nvme0c0n1.out‚93351.75244140625# proc.memory.rssMB‚ 2942.55078125* disk./.usagePercent‚41.964500029164384 disk.sda.in‚ 1.55078125 disk.dm-0.in‚ 24526.3671875* proc.memory.percent‚17.959904670715332 cpu‚25.541745914279506 disk.sda.out‚370.625 disk.sda1.in‚0 disk.sda3.out‚370.625 disk.dm-1.in‚ 1.55078125 disk.dm-1.out‚370.625 disk.sda3.in‚ 1.55078125 proc.cpu.threads‚17 network.recv‚ 2.125433e+06# disk.nvme0c0n1.in‚ 24526.3671875# disk.dm-0.out‚93351.75244140625 disk.sda2.out‚0) proc.memory.availableMB‚ 2918.36328125 disk.rbd1.out‚ 22.41796875 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 24526.3671875 disk.sda2.in‚0$ memory_percent‚82.18772411346436 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0$ disk./.usageGB‚625.3013763427734& disk.nvme0n1.out‚93351.75244140625 disk.rbd0.out‚0 network.sent‚ 9.3273e+06 disk.drbd2165.in‚0ë¸(l”É‚(:‰ û®ëÒ“¡®É gpu.0.gpu‚96.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚161.964' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.3928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ïi}éÊ‚(:Þ û®ëÒ€ÿ¹Ë network.recv‚ 2.129032e+06$ disk./.usageGB‚625.3009490966797 disk.sda2.in‚0 disk.dm-0.in‚ 24777.609375( proc.memory.availableMB‚ 2919.1015625 proc.cpu.threads‚17 disk.drbd2165.in‚0 network.sent‚ 9.334451e+06 disk.rbd1.in‚0 disk.rbd1.out‚ 22.55859375" disk.nvme0c0n1.in‚ 24777.609375* disk./.usagePercent‚41.964471356320544 disk.sda2.out‚0 disk.sda.out‚ 370.89453125& disk.nvme0c0n1.out‚93367.904296875 disk.nvme0n1.in‚ 24777.609375 disk.drbd2165.out‚0 cpu‚25.542130108070758 disk.dm-1.out‚ 370.89453125! disk.dm-0.out‚93367.904296875 disk.sda1.in‚0$ memory_percent‚82.18321800231934* proc.memory.percent‚17.959904670715332 disk.dm-1.in‚ 1.55078125 disk.sda.in‚ 1.55078125$ disk.nvme0n1.out‚93367.904296875 disk.sda1.out‚0 disk.sda3.in‚ 1.55078125 disk.sda3.out‚ 370.89453125 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2942.55078125 ySžË‚(:“ Š¯ëÒœäîÇ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚251.092' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.21840000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇV<)óÌ‚(:è Š¯ëÒ¢äÉ disk.sda.in‚ 1.55078125$ disk.nvme0c0n1.in‚24778.44140625( disk.nvme0c0n1.out‚93376.04541015625 network.sent‚ 9.33758e+06 disk.sda3.in‚ 1.55078125 disk.rbd1.out‚ 22.55859375 disk.dm-1.out‚ 371.26953125) proc.memory.availableMB‚ 2919.10546875 disk.rbd1.in‚0$ memory_percent‚82.18319416046143( disk./.usagePercent‚41.9644956770363 disk.dm-0.in‚24778.44140625# disk.dm-0.out‚93376.04541015625 disk.sda2.out‚0& disk.nvme0n1.out‚93376.04541015625 disk.sda1.out‚0 disk.sda3.out‚ 371.26953125 disk.dm-1.in‚ 1.55078125 cpu‚25.542419133160006 proc.cpu.threads‚17$ disk./.usageGB‚625.3013114929199 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚ 2.132413e+06" disk.nvme0n1.in‚24778.44140625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2942.55078125* proc.memory.percent‚17.959904670715332 disk.sda2.in‚0 disk.sda.out‚ 371.26953125ÇÒwžÍ‚(:“ ™¯ëÒé˜ÐÇ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚264.846' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.96920000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹¬ÊÏó΂(:è ™¯ëÒèïÓÉ disk.rbd1.out‚ 22.62109375 disk.sda.out‚ 371.56640625$ memory_percent‚82.18626976013184( proc.memory.availableMB‚ 2918.6015625 network.recv‚ 2.135883e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 371.56640625 disk.dm-0.in‚24778.44140625 disk.sda1.out‚0# proc.memory.rssMB‚ 2942.55078125# disk.dm-0.out‚93386.53173828125" disk.nvme0n1.in‚24778.44140625 disk.dm-1.in‚ 1.55078125 disk.dm-1.out‚ 371.56640625 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚93386.53173828125 network.sent‚ 9.34244e+06 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 1.55078125* proc.memory.percent‚17.959904670715332 cpu‚25.542726890842513) disk./.usagePercent‚41.96474298031438$ disk./.usageGB‚625.3049964904785 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚24778.44140625 proc.cpu.threads‚17& disk.nvme0n1.out‚93386.53173828125 disk.sda3.in‚ 1.55078125èZ÷¸2Ï- _wandbruntime‚8495  _runtime‚8495i¢°ÆÐÀ trainer/global_step‚4849 $ _timestamp‚1.784338334757807e+09 _runtime‚8495.342948553 _step‚396 & lr-AdamW/pg1‚0.0004236915761769519 & lr-AdamW/pg2‚0.0004236915761769519 ›‰ËÑÅ trainer/global_step‚4849 $ _timestamp‚1.784338334757807e+09 _runtime‚8495.342948553 _step‚396 & lr-AdamW/pg1‚0.0004236915761769519 & lr-AdamW/pg2‚0.0004236915761769519Œr^2Ò- _wandbruntime‚8495  _runtime‚8495 f>`$Ó÷O ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42002053 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 _step‚397 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1398278264077121 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26223699 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13778934 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1393724019748864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12444770 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1394326820503552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1398888846084096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4563397143552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83399569 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376578175569920 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1394326794330112 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4563397143552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376578543661056 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1394326794330112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1393724045176832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13779196 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1398279634105411 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1398888846084096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41387543 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚382458119650816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42012026 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42002053 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4562051753984 PCDeviceStatsMonitor.on_train_batch_start/al`_rÍlocation.large_pool.freed‚41387537 _runtime‚8495.344274286 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4562051753984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83389590 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4554244328257 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 trainer/global_step‚4849¹8ê©(ÔüO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚382458522278400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5879978617344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1398278264077121 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26223699 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13778934 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1393724019748864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12444770 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1394326820503552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1398888846084096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4563397143552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83399569 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376578175569920 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1394326794330112 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4563397143552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376578543661056 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1394326794330112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1393724045176832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13779196 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1398279634105411 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1398888846084096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41387543 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚382458119650816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42012026 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42002053 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4562051753984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41387537 _runtime‚8495.344274286 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4562051753984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83389590 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4554244328257 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 trainer/global_step‚4849 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42002053 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 _step‚397 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13781960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚398 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42010531 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4555167794503 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83406274 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1399167771849728 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5881137837568 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1399169143419904 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4562976789504 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1399169143419904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1394604821233664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41395749 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚382537202435584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1398558439349945 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4562976789504 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376656030068736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41395743 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83416267 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83406274 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚8496.741538796 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4564322186240 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1394604821233664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4564322186240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26229020 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1394604795060224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83416267 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12447308 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280¥³Ù'׈N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1394604821233664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4564322186240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26229020 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1394604795060224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83416267 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12447308 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13781960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚398 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42010531 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4555167794503 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83406274 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1399167771849728 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5881137837568 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1399169143419904 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4562976789504 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1399169143419904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1394604821233664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41395749 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚382537202435584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1398558439349945 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4562976789504 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376656030068736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41395743 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83416267 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83406274 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚8496.741538796 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4564322186240Ž[‚ 2Ø- _wandbruntime‚8496  _runtime‚8496X0+ Ùƒ 0loss/FRt1/assignment_loss‚1.6322475049758642 2loss/SRqqt1/assignment_loss‚0.5246051905183367 /loss/FRt1/detection_loss‚0.4891521645832478 epoch‚12 trainer/global_step‚4849 _runtime‚8496.824427295 _step‚399 3loss/FBt2/assignment_loss‚0.0016857758527623174 1loss/SRqqt1/detection_loss‚0.7263174219195931 1loss/SRqqt2/detection_loss‚0.7256773230675639 .loss/FBt1/detection_loss‚0.537063693360831 /loss/FBt2/detection_loss‚0.5918194147896894 % _timestamp‚1.7843383362396557e+09 2loss/SRqqt2/assignment_loss‚0.5018981778974242 1loss/FBt1/assignment_loss‚0.00749556331451233 /loss/FRt2/detection_loss‚1.6532472267555884 %loss/total_loss‚8.231278194548393 0loss/FRt2/assignment_loss‚0.84006873751297971¥Z÷¥Úˆ 0loss/FRt1/assignment_loss‚1.6322475049758642 2loss/SRqqt1/assignment_loss‚0.5246051905183367 /loss/FRt1/detection_loss‚0.4891521645832478 epoch‚12 trainer/global_step‚4849 _runtime‚8496.824427295 _step‚399 3loss/FBt2/assignment_loss‚0.0016857758527623174 1loss/SRqqt1/detection_loss‚0.7263174219195931 1loss/SRqqt2/detection_loss‚0.7256773230675639 .loss/FBt1/detection_loss‚0.537063693360831 /loss/FBt2/detection_loss‚0.5918194147896894 % _timestamp‚1.7843383362396557e+09 2loss/SRqqt2/assignment_loss‚0.5018981778974242 1loss/FBt1/assignment_loss‚0.00749556331451233 /loss/FRt2/detection_loss‚1.65324R¨5i72267555884 %loss/total_loss‚8.231278194548393 0loss/FRt2/assignment_loss‚0.8400687375129797ðrÂäÛ‚(:Ù ¨¯ëÒëˆîÌ# proc.memory.rssMB‚ 2942.55078125* proc.memory.percent‚17.959904670715332 disk.dm-1.out‚ 371.921875( disk.nvme0c0n1.out‚93496.88037109375( disk./.usagePercent‚41.9714977391053 disk.rbd1.in‚0 cpu‚25.543159183201684! disk.nvme0c0n1.in‚ 24779.40625# disk./.usageGB‚625.405647277832 disk.sda3.out‚ 371.921875 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 9.360385e+06 disk.sda1.in‚0# disk.dm-0.out‚93496.88037109375 disk.rbd0.in‚0 disk.dm-1.in‚ 1.55078125 disk.nvme0n1.in‚ 24779.40625 disk.sda2.in‚0 disk.rbd1.out‚ 22.62109375& disk.nvme0n1.out‚93496.88037109375 disk.dm-0.in‚ 24779.40625$ memory_percent‚82.18295574188232) proc.memory.availableMB‚ 2919.14453125 network.recv‚ 2.139346e+06 disk.sda1.out‚0 disk.sda3.in‚ 1.55078125 disk.sda.in‚ 1.55078125 disk.sda.out‚ 371.921875 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0{¿%3”Ü‚(:‰ ¨¯ëÒ£‡þÌ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚157.279' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„šB”Ý‚(:‰ ·¯ëÒîãæÇ gpu.0.gpu‚89.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î´äÞ‚(:Ù ·¯ëÒŽ×ñÉ( disk./.usagePercent‚41.9714962030601" disk.nvme0c0n1.in‚ 24861.171875 cpu‚25.543394562965815# memory_percent‚82.1845293045044) proc.memory.availableMB‚ 2918.88671875 network.sent‚ 9.410232e+06 disk.sda1.out‚0 disk.dm-1.out‚ 372.4296875 disk.rbd1.in‚0& disk.nvme0c0n1.out‚93502.431640625$ disk./.usageGB‚625.4056243896484$ disk.nvme0n1.out‚93502.431640625 disk.sda2.in‚0 disk.sda.in‚ 1.55078125 disk.drbd2165.out‚0 network.recv‚ 2.144117e+06 disk.nvme0n1.in‚ 24861.171875 disk.sda3.in‚ 1.55078125 disk.sda.out‚ 372.4296875! disk.dm-0.out‚93502.431640625* proc.memory.percent‚17.960286140441895 disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 1.55078125 disk.sda2.out‚0 disk.rbd1.out‚ 22.69921875 disk.dm-0.in‚ 24861.171875# proc.memory.rssMB‚ 2942.61328125 proc.cpu.threads‚17 disk.sda3.out‚ 372.4296875 disk.rbd0.in‚0 disk.drbd2165.in‚0.Åü”ß‚(:‰ ƯëÒîçÈÇ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñzæà‚(:Û ƯëÒ€ÞÆÉ disk.drbd2165.out‚0 network.sent‚ 9.415197e+06 disk.dm-1.out‚ 372.73046875 disk.sda.in‚ 1.55078125 disk.sda3.out‚ 372.73046875( disk./.usagePercent‚41.9714962030601 disk.rbd0.in‚0) proc.memory.availableMB‚ 2918.38671875 cpu‚25.54370028775605 network.recv‚ 2.147438e+06 disk.sda2.out‚0 disk.dm-0.in‚ 24861.609375$ disk./.usageGB‚625.4056243896484 disk.rbd1.in‚0 disk.rbd0.out‚0# memory_percent‚82.1875810623169 disk.sda.out‚ 372.73046875 disk.dm-1.in‚ 1.55078125" disk.nvme0c0n1.in‚ 24861.609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 22.69921875& disk.nvme0c0n1.out‚93509.658203125$ disk.nvme0n1.out‚93509.658203125 proc.cpu.threads‚17 disk.drbd2165.in‚0! disk.dm-0.out‚93509.658203125 disk.nvme0n1.in‚ 24861.609375 disk.sda3.in‚ 1.55078125# proc.memory.rssMB‚ 2942.61328125* proc.memory.percent‚17.960286140441895|õÅžá‚(:“ Õ¯ëÒ½Ü·É gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚272.86' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.571999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚007·Ëëâ‚(:à Õ¯ëÒÄÉíË disk.sda2.in‚0 disk.sda.in‚ 1.55078125 disk.sda3.in‚ 1.55078125# proc.memory.rssMB‚ 2942.61328125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd0.in‚0$ memory_percent‚88.11562061309814 disk.rbd1.out‚ 22.76171875$ disk.nvme0c0n1.in‚24862.57421875 disk.dm-0.in‚24862.57421875# disk.nvme0n1.out‚93519.66796875 disk.sda1.out‚0 network.sent‚ 9.420162e+06$ disk./.usageGB‚625.3945426940918 disk.dm-1.in‚ 1.55078125 disk.dm-1.out‚ 373.56640625 disk.sda3.out‚ 373.56640625 disk.rbd0.out‚0 cpu‚25.54404686713261) disk./.usagePercent‚41.97075250117313 disk.drbd2165.in‚0 disk.dm-0.out‚93519.66796875* proc.memory.percent‚17.960286140441895% disk.nvme0c0n1.out‚93519.66796875" disk.nvme0n1.in‚24862.57421875 disk.sda1.in‚0) proc.memory.availableMB‚ 1947.13671875 network.recv‚ 2.151045e+06 disk.sda.out‚ 373.56640625 disk.drbd2165.out‚0 disk.sda2.out‚0]L“ã‚(:ˆ ä¯ëÒÜÑ¿Ç gpu.0.gpu‚89.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚191.175' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.235 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ žÜä‚(:Ñ ä¯ëÒ‘¾ËÉ disk.rbd0.out‚0 disk.sda2.out‚0 network.sent‚ 9.425271e+06 disk.sda3.in‚ 1.55078125( proc.memory.availableMB‚ 1946.7421875* proc.memory.percent‚17.960286140441895 disk.sda3.out‚ 374.0390625) disk./.usagePercent‚41.97075659729368$ disk.nvme0c0n1.in‚24862.57421875 disk.dm-0.in‚24862.57421875 disk.nvme0n1.out‚ 93522.46875 disk.rbd0.in‚0 disk.drbd2165.out‚0 cpu‚25.54433324944053 disk.sda2.in‚0 disk.sda.in‚ 1.55078125 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.dm-0.out‚ 93522.46875" disk.nvme0n1.in‚24862.57421875 disk.sda1.in‚0 disk.dm-1.out‚ 374.0390625# disk./.usageGB‚625.394603729248" disk.nvme0c0n1.out‚ 93522.46875 disk.dm-1.in‚ 1.55078125 disk.rbd1.out‚ 22.8671875$ memory_percent‚88.11802864074707# proc.memory.rssMB‚ 2942.61328125 network.recv‚ 2.154629e+06 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 374.0390625ãþym2å- _wandbruntime‚8575  _runtime‚8575ã´¥4Éæà ' lr-AdamW/pg1‚0.00042242096419426486 ' lr-AdamW/pg2‚0.00042242096419426486 trainer/global_step‚4899 % _timestamp‚1.7843384150979955e+09 _runtime‚8575.683083334 _step‚400µÈÚÎçÈ % _timestamp‚1.7843384150979955e+09 _runtime‚8575.683083334 _step‚400 ' lr-AdamW/pg1‚0.00042242096419426486 ' lr-AdamW/pg2‚0.00042242096419426486 trainer/global_step‚4899a 52è- _wandbruntime‚8575  _runtime‚8575¦`«þ'éøO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 $ _timestamp‚1.784338415099146e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41797843 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1407618134085632 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26490367 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84225642 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13918452 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚386392004883968 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1408226830835712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4609713074176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5939545563648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚380452495953920 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12571920 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42427805 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4608367684608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41797843 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1412835198520320 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1412835198520320 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1412836570083328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4608367684608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42427805 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1408226857009152 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚401 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42437778 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1407618108657664 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41797837 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1412836570083328 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12571915 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4609713074176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4601826041687 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚8575.684424827 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26490638 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1412218590099029 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 trainer/global_step‚4899 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42437778 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13918718 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5939508930048 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚380452864045056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚386392409608704 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1408226830835712 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1412219960127319 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995WÜfï(êýO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚380452864045056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚386392409608704 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1408226830835712 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1412219960127319 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 $ _timestamp‚1.784338415099146e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41797843 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1407618134085632 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26490367 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84225642 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13918452 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚386392004883968 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1408226830835712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4609713074176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5939545563648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚380452495953920 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12571920 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42427805 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4608367684608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41797843 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1412835198520320 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1412835198520320 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1412836570083328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4608367684608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42427805 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1408226857009152 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚401 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42437778 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1407618108657664 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41797837 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1412836570083328 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12571915 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4609713074176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4601826041687 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚8575.684424827 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26490638 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1412218590099029 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 trainer/global_step‚4899 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42437778 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13918718 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5939508930048 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820‘µjl}2ë- _wandbruntime‚8577  _runtime‚8577¸|‚ìƒN UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1408504831565824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84252319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380530350452736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1412497395343451 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4899 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1408504857739264 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13921472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13921205 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42436283 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26495663 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1407896015863808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 VIDeviceStatsMonito˜ù€W‡r.on_train_batch_end/inactive_split.large_pool.allocated‚12574463 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚386471084153344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1413114124285952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84242326 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4601404907611 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4609292720128 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1408504857739264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚402 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1408504831565824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4609292720128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12574458 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚386470679435776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380529982361600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5940697074176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4602749508045 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1412498765371853 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26495935 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42446270 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4601404907611 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4609292720128 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1408504857739264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚402 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1408504831565824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4609292720128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12574458 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚386470679435776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380529982361600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5940697074176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4602749508045 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1412498765371853 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26495935 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42446270 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84252319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380530350452736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1412497395343451 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4899 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1408504857739264 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13921472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13921205 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42436283 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26495663 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1407896015863808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12574463 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚386471084153344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1413114124285952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995’)óÇ¢2î- _wandbruntime‚8577  _runtime‚8577Ú4“Jï€ 0loss/FBt2/assignment_loss‚0.0715434354488317 2loss/SRqqt2/assignment_loss‚0.4756708358375719 .loss/FRt1/detection_loss‚0.555734564744715 1loss/SRqqt1/detection_loss‚0.6879552236643254 1loss/SRqqt2/detection_loss‚0.6766185719649757 %loss/total_loss‚8.080467212046926 _runtime‚8577.161063559 3loss/SRqqt1/assignment_loss‚0.45519748484852884 epoch‚12 _step‚403 0loss/FRt2/assignment_loss‚0.7972713544402482 2loss/FBt1/assignment_loss‚0.007566801075287084 .loss/FRt2/detection_loss‚1.623901580048523 .loss/FBt1/detection_loss‚0.577709571002935 /loss/FBt2/detection_loss‚0.5025889113959339 trainer/global_step‚4899 % _timestamp‚1.7843384165762885e+09 0loss/FRt1/assignment_loss‚1.6487088775750502¹ñåÏ ð… trainer/global_step‚4899 % _timestamp‚1.7843384165762885e+09 0loss/FRt1/assignment_loss‚1.6487088775750502 0loss/FBt2/assignment_loss‚0.0715434354488317 2loss/SRqqt2/assignment_loss‚0.4756708358375719 .loss/FRt1/detection_loss‚0.555734564744715 1loss/SRqqt1/detection_loss‚0.6879552236643254 1loss/SRqqt2/detection_loss‚0.6766185719649757 %loss/total_loss‚8.080467212046926 _runtime‚8577.161063559 3loss/SRqqt1/assignment_loss‚0.45519748484852884 epoch‚12 _step‚403 0loss/FRt2/assignment_loss‚0.7972713544402482 2loss/FBt1/assignment_loss‚0.007566801075287084 .loss/FRt2/detection_loss‚1.623901580048523 .loss/FBt1/detection_loss‚0.577709571002935 /loss/FBt2/detection_loss‚0.5025889113959339“Bêžñ‚(:“ ó¯ëÒ÷æ¹È gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚289.743' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.94859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07™áIîò‚(:ã ó¯ëÒð¨¾É disk.sda3.in‚ 1.55078125 disk.drbd2165.in‚0 disk.sda1.in‚0 network.sent‚ 9.430253e+06 disk.rbd0.out‚0 disk.sda3.out‚ 374.37109375 disk.sda2.out‚0' disk.nvme0c0n1.out‚93534.2177734375 disk.dm-0.in‚24950.23828125# proc.memory.rssMB‚ 2942.61328125 disk.sda.out‚ 374.37109375$ disk./.usageGB‚625.3946800231934 disk.dm-1.out‚ 374.37109375 disk.rbd1.in‚0$ disk.nvme0c0n1.in‚24950.23828125% disk.nvme0n1.out‚93534.2177734375 disk.dm-1.in‚ 1.55078125$ memory_percent‚88.11798095703125# proc.memory.availableMB‚1946.75 disk.sda.in‚ 1.55078125) disk./.usagePercent‚41.97076171744436 disk.rbd0.in‚0 disk.rbd1.out‚23.04833984375 disk.sda1.out‚0 cpu‚25.54457614405111 proc.cpu.threads‚17 network.recv‚ 2.158136e+06 disk.sda2.in‚0 disk.drbd2165.out‚0" disk.dm-0.out‚93534.2177734375" disk.nvme0n1.in‚24950.23828125* proc.memory.percent‚17.960286140441895ªõŠ=Ÿó‚(:” ‚°ëÒÍÍÉ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚187.686' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.537200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´{¨Pïô‚(:ä ‚°ëÒŸÏÛË) disk./.usagePercent‚41.97079320637107 disk.drbd2165.out‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 1946.703125( disk.nvme0c0n1.out‚93542.10791015625 disk.sda1.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2942.61328125 disk.drbd2165.in‚0# disk.dm-0.out‚93542.10791015625 network.sent‚ 9.492973e+06 disk.rbd1.in‚0 disk.sda.out‚ 374.66796875 disk.rbd0.in‚0 disk.dm-1.out‚ 374.66796875$ memory_percent‚88.11826705932617 network.recv‚ 2.162315e+06 disk.sda1.in‚0* proc.memory.percent‚17.960286140441895# disk./.usageGB‚625.395149230957 disk.rbd1.out‚23.04833984375 disk.dm-0.in‚ 24951.203125 disk.nvme0n1.in‚ 24951.203125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 24951.203125 disk.sda3.in‚ 1.55078125 disk.sda3.out‚ 374.66796875 cpu‚25.544919801077945 disk.sda.in‚ 1.55078125& disk.nvme0n1.out‚93542.10791015625 disk.dm-1.in‚ 1.55078125 proc.cpu.threads‚17ðβžõ‚(:“ ‘°ëÒòþúÇ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚291.384' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.27680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çš® ëö‚(:à ‘°ëÒ·¢ùÉ network.sent‚ 9.498e+06$ memory_percent‚88.11993598937988) disk./.usagePercent‚41.97078014998682 disk.sda2.in‚0 disk.sda.in‚ 1.55078125 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚93551.17333984375 disk.sda3.in‚ 1.55078125 disk.sda3.out‚ 374.94140625 disk.rbd0.in‚0 disk.rbd1.out‚ 23.2802734375 disk.sda.out‚ 374.94140625# disk.dm-0.out‚93551.17333984375( proc.memory.availableMB‚ 1946.4296875 cpu‚25.54520014511648 disk.nvme0n1.in‚ 24951.203125& disk.nvme0n1.out‚93551.17333984375 disk.sda2.out‚0 disk.dm-0.in‚ 24951.203125 proc.cpu.threads‚17 network.recv‚ 2.16577e+06$ disk./.usageGB‚625.3949546813965 disk.sda1.out‚0 disk.dm-1.in‚ 1.55078125" disk.nvme0c0n1.in‚ 24951.203125 disk.rbd0.out‚0 disk.rbd1.in‚0* proc.memory.percent‚17.960286140441895 disk.sda1.in‚0 disk.dm-1.out‚ 374.94140625 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2942.613281259@=”÷‚(:‰  °ëÒøÏ€È gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚263.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|§¸íø‚(:â  °ëÒ¬—×É disk.drbd2165.in‚0 disk.dm-0.in‚24952.03515625 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 1.55078125 disk.rbd1.out‚ 23.2802734375# proc.memory.rssMB‚ 2942.61328125$ disk.nvme0c0n1.in‚24952.03515625 disk.sda3.out‚ 375.1640625* proc.memory.percent‚17.960286140441895" disk.nvme0n1.in‚24952.03515625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.in‚ 1.55078125 disk.rbd1.in‚0$ memory_percent‚88.12167644500732) proc.memory.availableMB‚ 1946.14453125$ disk./.usageGB‚625.4688720703125 disk.sda.out‚ 375.1640625! disk.dm-0.out‚93712.482421875 disk.rbd0.in‚0 disk.sda.in‚ 1.55078125 disk.sda2.in‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚93712.482421875$ disk.nvme0n1.out‚93712.482421875 disk.sda1.in‚0 disk.dm-1.out‚ 375.1640625 network.recv‚ 2.169232e+06( disk./.usagePercent‚41.9757408079778 cpu‚25.545499099749595 network.sent‚ 9.503092e+06ÆÂã|”ù‚(:‰ ¯°ëÒŒÈ³É gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.766' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.7532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#‘ æú‚(:Û ¯°ëÒµŠßË network.recv‚ 2.172635e+06! disk.nvme0n1.in‚ 25046.6328125 disk.rbd0.in‚0 disk.dm-1.in‚ 1.55078125 disk.rbd1.in‚0* proc.memory.percent‚17.960286140441895$ disk./.usageGB‚625.5051612854004 cpu‚25.545869300749853 disk.sda3.in‚ 1.55078125 disk.rbd0.out‚0 disk.dm-1.out‚ 375.88671875 proc.cpu.threads‚17" disk.nvme0n1.out‚ 93759.3515625 disk.sda.out‚ 375.88671875# disk.nvme0c0n1.in‚ 25046.6328125( proc.memory.availableMB‚ 1946.5703125 disk.sda.in‚ 1.55078125 disk.drbd2165.out‚0 disk.dm-0.in‚ 25046.6328125 disk.dm-0.out‚ 93759.3515625 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 23.3583984375$ memory_percent‚88.11907768249512) disk./.usagePercent‚41.97817620765103 disk.sda1.out‚0$ disk.nvme0c0n1.out‚ 93759.3515625# proc.memory.rssMB‚ 2942.61328125 network.sent‚ 9.51012e+06 disk.sda3.out‚ 375.88671875 disk.drbd2165.in‚0”É©'žû‚(:“ ¾°ëÒ‰ö¦Ç gpu.0.gpu‚89.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚258.285' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.65700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Âïïü‚(:ä ¾°ëÒ⦇Ê disk.drbd2165.out‚0! disk.nvme0n1.in‚ 25046.6328125 disk.rbd0.out‚0# proc.memory.rssMB‚ 2942.61328125 cpu‚25.546117157312583 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 25046.6328125# memory_percent‚88.1237268447876 network.sent‚ 9.515087e+06 disk.rbd1.out‚ 23.3583984375 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚93762.29736328125 disk.dm-0.in‚ 25046.6328125 disk.sda.in‚ 1.55078125 disk.sda.out‚ 376.2109375& disk.nvme0n1.out‚93762.29736328125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 376.2109375 disk.sda3.out‚ 376.2109375) proc.memory.availableMB‚ 1945.80859375 disk.rbd0.in‚0 network.recv‚ 2.1758e+06$ disk./.usageGB‚626.3658180236816 disk.sda3.in‚ 1.55078125 proc.cpu.threads‚17 disk.rbd1.in‚0# disk.dm-0.out‚93762.29736328125* proc.memory.percent‚17.960286140441895* disk./.usagePercent‚42.035935603496185 disk.dm-1.in‚ 1.55078125#M2ý- _wandbruntime‚8656  _runtime‚8656Öø¸RÇþÁ & lr-AdamW/pg2‚0.0004211387150847387 trainer/global_step‚4949 % _timestamp‚1.7843384954717827e+09 _runtime‚8656.056876206 _step‚404 & lr-AdamW/pg1‚0.0004211387150847387É¥ÈÌÿÆ _runtime‚8656.056876206 _step‚404 & lr-AdamW/pg1‚0.0004211387150847387 & lr-AdamW/pg2‚0.0004211387150847387 trainer/global_step‚4949 % _timestamp‚1.7843384954717827e+09”¹"m±2€- _wandbruntime‚8656  _runtime‚86560ælùO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1426781550956544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14057558 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4654683615232 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42208143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚4949 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4656029004800 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42208137 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5998839559680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42853557 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1426782922519552 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1422126893514752 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1422126893514752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42863530 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚405 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1421512222994432 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _runtime‚8656.061831222 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42208143 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4656029004800 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85061694 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚384327184429056 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42853557 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42863530 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1426782922519552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390326023988736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12699065 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26756895 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1421512197566464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5998802926080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1426160286149227 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1426158916120937£à†E(‚þO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚405 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1421512222994432 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _runtime‚8656.061831222 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42208143 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4656029004800 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85061694 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚384327184429056 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42853557 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42863530 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1426782922519552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390326023988736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12699065 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26756895 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1421512197566464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5998802926080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1426160286149227 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1426158916120937 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1426781550956544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14057558 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4654683615232 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42208143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚4949 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4656029004800 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42208137 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5998839559680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42853557 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1426782922519552 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1422126893514752 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1422126893514752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42863530•^”é2ƒ- _wandbruntime‚8657  _runtime‚8657ˆ%ÊY'„‚N K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26761926 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4656954047488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚384404302745600 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14060318 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1422404868071424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1426439091393761 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1422404894244864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85088371 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42216343 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390404285581824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚384404670836736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5999982836224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42216343 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42216349 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1426437721365359 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4648986621153 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1427061848292352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _step‚406 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390404690299392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚8657.455450624 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42862035 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1427060476722176 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1427061848292352 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12701613 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42862035 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85078378 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1421790079344640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85088371 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1422404868071424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4656954047488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42216349 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42872022 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1421790104772608 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26761926 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4656954047488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚384404302745600 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14060318 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1422404868071424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1426439091393761 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1422404894244864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85088371 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42216343 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390404285581824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚384404670836736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5999982836224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42216343 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42216349 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1426437721365359 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4648986621153 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1427061848292352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _step‚406 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390404690299392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚8657.455450624 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42862035 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1427060476722176 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1427061848292352 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12701613 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42862035 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85078378 TFDeviceStatsMonitor.on_train_batch_end¼Ç×/inactive_split_bytes.all.current‚ 404717568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1421790079344640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85088371 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1422404868071424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4656954047488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42216349 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42872022 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097–Ç?62†- _wandbruntime‚8657  _runtime‚8657O¾—‡þ % _timestamp‚1.7843384969519958e+09 0loss/FRt1/assignment_loss‚1.6795810538648912 0loss/FBt1/assignment_loss‚0.0704768327796062 /loss/FRt1/detection_loss‚0.5142140304924594 /loss/FRt2/detection_loss‚1.6166082281012328 /loss/FBt1/detection_loss‚0.6052766535547562 _runtime‚ 8657.53680873 _step‚407 2loss/SRqqt1/assignment_loss‚0.5175951595871876 1loss/FBt2/assignment_loss‚0.14336252334446692 1loss/SRqqt2/detection_loss‚0.7763286068443588 /loss/FBt2/detection_loss‚0.5760535917710513 trainer/global_step‚4949 epoch‚12 0loss/FRt2/assignment_loss‚0.8480356826414517 2loss/SRqqt2/assignment_loss‚0.6337930265340204 1loss/SRqqt1/detection_loss‚0.7341836190448167 #loss/total_loss‚8.7155090085603@ÕÂk ˆƒ epoch‚12 0loss/FRt2/assignment_loss‚0.8480356826414517 2loss/SRqqt2/assignment_loss‚0.6337930265340204 1loss/SRqqt1/detection_loss‚0.7341836190448167 #loss/total_loss‚8.7155090085603 % _timestamp‚1.7843384969519958e+09 0loss/FRt1/assignment_loss‚1.6795810538648912 0loss/FBt1/assignment_loss‚0.0704768327796062 /loss/FRt1/detection_loss‚0.5142140304924594 /loss/FRt2/detection_loss‚1.6166082281012328 /loss/FBt1/detection_loss‚0.6052766535547562 _runtime‚ 8657.53680873 _step‚407 2loss/SRqqt1/assignment_loss‚0.5175951595871876 1loss/FBt2/assignment_loss‚0.14336252334446692 1loss/SRqqt2/detection_loss‚0.7763286068443588 /loss/FBt2/detection_loss‚0.5760535917710513 trainer/global_step‚4949—òN®i’‰‚(:‡ Í°ëÒœëÇ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚267.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m•ü_슂(:á Í°ëÒ‡¡¾É$ memory_percent‚88.12313079833984 network.sent‚ 9.51788e+06 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2165.in‚0$ disk./.usageGB‚628.2536315917969 disk.rbd1.out‚ 23.4208984375 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚25047.46484375# disk.dm-0.out‚95415.97412109375& disk.nvme0n1.out‚95415.97412109375 disk.dm-1.in‚ 1.55078125 disk.dm-1.out‚ 376.578125 disk.rbd1.in‚0 cpu‚25.54635738414041& proc.memory.availableMB‚ 1945.90625 disk.rbd0.in‚0( disk.nvme0c0n1.out‚95415.97412109375 disk.dm-0.in‚25047.46484375 disk.sda1.out‚0 disk.sda3.in‚ 1.55078125 network.recv‚ 2.178411e+06 disk.sda.in‚ 1.55078125 disk.drbd2165.out‚0" disk.nvme0n1.in‚25047.46484375 disk.sda1.in‚0* proc.memory.percent‚17.960286140441895) disk./.usagePercent‚42.16262835603349# proc.memory.rssMB‚ 2942.61328125 disk.sda2.out‚0 disk.sda.out‚ 376.578125 disk.sda3.out‚ 376.578125ä2}=Ÿ‹‚(:” Ü°ëÒËøéÊ gpu.0.gpu‚44.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚193.067' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚38.613400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}¨ÛƒíŒ‚(:â Ü°ëÒÆÎÃÍ* proc.memory.percent‚17.960286140441895 network.sent‚ 9.582026e+06) disk./.usagePercent‚42.29918303080479# disk.nvme0c0n1.in‚ 25048.4296875 disk.sda2.out‚0 network.recv‚ 2.182782e+06 disk.sda.out‚ 376.89453125 disk.sda1.out‚0 disk.rbd1.out‚ 23.4208984375 disk.dm-1.out‚ 376.89453125 proc.cpu.threads‚17& disk.nvme0c0n1.out‚95499.220703125 disk.sda1.in‚0 disk.sda3.in‚ 1.55078125 disk.sda2.in‚0# proc.memory.rssMB‚ 2942.61328125 disk.drbd2165.out‚0 disk.dm-1.in‚ 1.55078125 disk.rbd1.in‚0 disk.sda.in‚ 1.55078125 disk.drbd2165.in‚0 disk.dm-0.in‚ 25048.4296875 disk.sda3.out‚ 376.89453125 disk.rbd0.in‚0( proc.memory.availableMB‚ 1946.1328125 cpu‚25.546712014599727$ memory_percent‚88.12174797058105$ disk./.usageGB‚630.2883949279785! disk.dm-0.out‚95499.220703125! disk.nvme0n1.in‚ 25048.4296875$ disk.nvme0n1.out‚95499.220703125 disk.rbd0.out‚0ÀaüHž‚(:“ ë°ëÒüü†Ç gpu.0.gpu‚86.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚292.743' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.54859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ×⎂(:× ë°ëÒÁØþÉ disk.sda3.out‚ 377.3671875 disk.sda.in‚1.96875 network.sent‚ 9.587011e+06* proc.memory.percent‚17.960286140441895" disk.nvme0n1.in‚25117.66796875 disk.rbd0.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2942.61328125 disk.drbd2165.out‚0" disk.dm-0.out‚96522.0966796875 disk.rbd1.out‚ 23.4990234375 disk.sda.out‚ 377.3671875$ disk.nvme0c0n1.in‚25117.66796875 disk.sda3.in‚1.96875 disk.sda2.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚96522.0966796875 network.recv‚ 2.186472e+06$ disk./.usageGB‚632.1820526123047 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 377.3671875$ memory_percent‚88.12179565429688 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.in‚1.96875$ proc.memory.availableMB‚1946.125 cpu‚25.54696853032716 disk.sda1.out‚0) disk./.usagePercent‚42.42626798688455' disk.nvme0c0n1.out‚96522.0966796875 disk.dm-0.in‚25117.66796875®Å{Ÿ”‚(:‰ ú°ëÒûÌÕÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚286.356' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.2712 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L)kæà‚(:Õ ú°ëÒòäØÉ$ memory_percent‚88.12079429626465 network.sent‚ 9.589933e+06 disk.sda3.out‚ 378.296875* proc.memory.percent‚17.960286140441895 disk.dm-1.in‚1.96875 disk.dm-1.out‚ 378.296875$ disk.nvme0n1.out‚98757.376953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 1946.2890625# proc.memory.rssMB‚ 2942.61328125 proc.cpu.threads‚17 disk.rbd1.out‚ 23.4990234375 disk.sda.out‚ 378.296875$ disk.nvme0c0n1.in‚25133.34765625& disk.nvme0c0n1.out‚98757.376953125 disk.rbd0.in‚0 disk.sda2.in‚0 network.recv‚ 2.189695e+06! disk.dm-0.out‚98757.376953125 cpu‚25.54726156814033 disk.drbd2165.in‚0 disk.dm-0.in‚25133.34765625 disk.sda3.in‚1.96875 disk.sda2.out‚0) disk./.usagePercent‚42.55097693703902 disk.drbd2165.out‚0$ disk./.usageGB‚634.0403060913086 disk.rbd1.in‚0 disk.sda.in‚1.96875" disk.nvme0n1.in‚25133.34765625ð™¤Õ“‘‚(:ˆ ‰±ëÒ†ÙšÊ gpu.0.gpu‚21.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚338.986' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.7972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=ÔÀ"ð’‚(:å ‰±ëÒâ««Ì) disk.nvme0c0n1.out‚100254.27099609375 network.sent‚ 9.594948e+06$ disk./.usageGB‚635.9841194152832 disk.sda.out‚ 378.92578125 disk.sda2.in‚0$ memory_percent‚88.11924457550049) proc.memory.availableMB‚ 1946.54296875 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 23.5615234375 disk.rbd0.in‚0$ disk.dm-0.out‚100254.27099609375 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚25133.44140625 disk.sda3.in‚1.96875 disk.rbd0.out‚0# proc.memory.rssMB‚ 2942.61328125 proc.cpu.threads‚17 disk.drbd2165.out‚0* proc.memory.percent‚17.960286140441895 cpu‚25.54762133651989 network.recv‚ 2.193171e+06) disk./.usagePercent‚42.68142788017897 disk.sda.in‚1.96875' disk.nvme0n1.out‚100254.27099609375 disk.sda1.out‚0 disk.sda3.out‚ 378.92578125 disk.dm-1.out‚ 378.92578125 disk.dm-0.in‚25133.44140625" disk.nvme0n1.in‚25133.44140625 disk.dm-1.in‚1.96875%ÆÕ2“- _wandbruntime‚8736  _runtime‚8736´µ Ç”Á & lr-AdamW/pg1‚0.0004198449112048377 & lr-AdamW/pg2‚0.0004198449112048377 trainer/global_step‚4999 % _timestamp‚1.7843385757150424e+09 _runtime‚8736.300143132 _step‚408ØüsÈÌ•Æ % _timestamp‚1.7843385757150424e+09 _runtime‚8736.300143132 _step‚408 & lr-AdamW/pg1‚0.0004198449112048377 & lr-AdamW/pg2‚0.0004198449112048377 trainer/global_step‚4999˜›¼m 2–- _wandbruntime‚8736  _runtime‚8736Ú3gþ'—øO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42618437 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27022570 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1440727903392768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6058054278656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43279309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42618443 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388201504813056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1435406311903232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1435406286475264 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43279309 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1440729274955776 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6058088815104 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85897746 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4702344935424 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85907725 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1436026930020352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚4999 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14196350 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1440729274955776 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43289282 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12826220 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4692955667581 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12826215 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1436026930020352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42618443 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4702344935424 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚394259191000576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1436026903846912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚394259593628160 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1440100612171135 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚409 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4694300267903 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1440727903392768 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43289282 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1436026903846912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4700999545856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995©¾(˜ýO IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1436026930020352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚4999 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14196350 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1440729274955776 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43289282 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12826220 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4692955667581 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12826215 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1436026930020352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42618443 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4702344935424 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚394259191000576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1436026903846912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚394259593628160 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1440100612171135 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚409 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4694300267903 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1440727903392768 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43289282 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1436026903846912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4700999545856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42618437 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27022570 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1440727903392768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6058054278656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43279309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42618443 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388201504813056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1435406311903232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1435406286475264 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43279309 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1440729274955776 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6058088815104 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85897746 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4702344935424 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85907725 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1436304930750464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27027575 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85914430 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6059278534144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.al­•b¬ l.peak‚ 47545124352 % _timestamp‚1.7843385771112542e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14199069 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4693879133827 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 _step‚410 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85924423 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85914430 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85924423 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4703269978112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14198817 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43297774 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43297774 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1436304930750464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚394337865037312 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12828763 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚ 8737.69649595 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1441008200728576 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1436304904577024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1440379417415669 trainer/global_step‚4999 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1441008200728576 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4695223734261 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42626649 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4701924581376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42626643 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43287787 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6059241907712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1440378047387267 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85924423 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85914430 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85924423 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4703269978112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14198817 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43297774 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43297774 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1436304930750464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚394337865037312 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12828763 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚ 8737.69649595 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1441008200728576 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1436304904577024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1440379417415669 trainer/global_step‚4999 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1441008200728576 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4695223734261 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42626649 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4701924581376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42626643 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43287787 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6059241907712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1440378047387267 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1436304930750464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27027575 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85914430 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6059278534144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 % _timestamp‚1.7843385771112542e+09š2‚+|2œ- _wandbruntime‚8737  _runtime‚8737è8ÿ 1loss/FBt2/assignment_loss‚0.02300126805948158 -loss/FBt2/detection_loss‚0.73806297974661 %loss/total_loss‚8.458589442200163 trainer/global_step‚4999 1loss/SRqqt2/detection_loss‚0.6955034999475006 _runtime‚8737.778962571 2loss/SRqqt2/assignment_loss‚0.5407068144685291 .loss/FRt2/detection_loss‚1.772603270418942 /loss/FBt1/detection_loss‚0.4223280557050001 0loss/FRt1/assignment_loss‚1.5919870796752618 2loss/SRqqt1/assignment_loss‚0.5155017148089035 1loss/FBt1/assignment_loss‚0.03890811798066225 /loss/FRt1/detection_loss‚0.5190035028462799 1loss/SRqqt1/detection_loss‚0.7931302354947941 epoch‚12 % _timestamp‚1.7843385771941888e+09 _step‚411 0loss/FRt2/assignment_loss‚0.8078529030481975`ë « ž„ 0loss/FRt1/assignment_loss‚1.5919870796752618 2loss/SRqqt1/assignment_loss‚0.5155017148089035 1loss/FBt1/assignment_loss‚0.03890811798066225 /loss/FRt1/detection_loss‚0.5190035028462799 1loss/SRqqt1/detection_loss‚0.7931302354947941 epoch‚12 % _timestamp‚1.7843385771941888e+09 _step‚411 0loss/FRt2/assignment_loss‚0.8078529030481975 1loss/FBt2/assignment_loss‚0.02300126805948158 -loss/FBt2/detection_loss‚0.73806297974661 %loss/total_loss‚8.458589442200163 trainer/global_step‚4999 1loss/SRqqt2/detection_loss‚0.6955034999475006 _runtime‚8737.778962571 2loss/SRqqt2/assignment_loss‚0.5407068144685291 .loss/FRt2/detection_loss‚1.772603270418942 /loss/FBt1/detection_loss‚0.4223280557050001›ºá딟‚(:‰ ˜±ëÒõȘÈ gpu.0.gpu‚96.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚235.703' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.1406 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t4þýÛ ‚(:Ð ˜±ëÒÀ‡ýÉ disk.sda2.in‚0' disk.nvme0c0n1.out‚102394.048828125" disk.dm-0.out‚102394.048828125% disk.nvme0n1.out‚102394.048828125 disk.rbd1.in‚0* disk./.usagePercent‚42.811032718418254 disk.drbd2165.in‚0 disk.sda3.in‚1.96875 disk.sda2.out‚0 disk.sda.in‚1.96875' proc.memory.availableMB‚ 1946.640625 network.recv‚ 2.196451e+06 disk.nvme0n1.in‚ 25134.3125 disk.sda1.in‚0$ memory_percent‚88.11864852905273# proc.memory.rssMB‚ 2942.61328125* proc.memory.percent‚17.960286140441895 cpu‚25.547896327418314 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 379.1328125 disk.drbd2165.out‚0 disk.rbd1.out‚ 23.5615234375 proc.cpu.threads‚17$ disk./.usageGB‚637.9153251647949 disk.dm-0.in‚ 25134.3125 disk.dm-1.in‚1.96875 disk.dm-1.out‚ 379.1328125 network.sent‚ 9.599913e+06 disk.nvme0c0n1.in‚ 25134.3125 disk.sda3.out‚ 379.1328125 disk.rbd0.in‚0,f!ñ¡‚(:æ §±ëÒžÌÅË disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0$ disk.dm-0.out‚104596.12646484375 disk.sda2.out‚0' disk.nvme0n1.out‚104596.12646484375 disk.sda1.out‚0 disk.sda3.in‚1.96875 disk.sda.in‚1.96875$ disk.nvme0c0n1.in‚25135.14453125* proc.memory.percent‚17.960286140441895$ disk./.usageGB‚639.8478660583496" disk.nvme0n1.in‚25135.14453125 disk.dm-1.in‚1.96875 disk.dm-1.out‚ 379.87890625 disk.rbd1.out‚ 23.6396484375 disk.drbd2165.out‚0* disk./.usagePercent‚42.940727159294525 disk.sda3.out‚ 379.87890625 disk.sda.out‚ 379.87890625 disk.dm-0.in‚25135.14453125# proc.memory.rssMB‚ 2942.61328125 disk.drbd2165.in‚0# memory_percent‚88.1188154220581 network.sent‚ 9.664726e+06 disk.sda1.in‚0 cpu‚25.548146535703808 network.recv‚ 2.201464e+06) disk.nvme0c0n1.out‚104596.12646484375 disk.sda2.in‚0) proc.memory.availableMB‚ 1946.61328125 proc.cpu.threads‚17­Ê#ÆŸ¢‚(:” §±ëÒÞ¾×Ë gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚280.116' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.023199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÖ̳’£‚(:‡ ¶±ëÒÛÖÈÉ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚336.177' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.2354 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘îˆí¤‚(:â ¶±ëÒ£·÷Ë disk.dm-1.in‚1.96875# proc.memory.rssMB‚ 2942.61328125$ disk.nvme0c0n1.in‚25233.47265625" disk.nvme0n1.in‚25233.47265625 disk.sda3.in‚1.96875 disk.sda2.in‚0$ memory_percent‚88.11888694763184* proc.memory.percent‚17.960286140441895$ disk./.usageGB‚639.9883270263672 disk.sda.in‚1.96875 disk.drbd2165.in‚0 disk.dm-0.in‚25233.47265625 disk.rbd0.out‚0( proc.memory.availableMB‚ 1946.6015625 cpu‚25.548459879189384 network.sent‚ 9.667749e+06 disk.dm-1.out‚ 380.30859375 disk.rbd1.out‚ 23.6396484375 disk.rbd1.in‚0 disk.sda.out‚ 380.30859375( disk.nvme0c0n1.out‚106211.5634765625 disk.sda1.in‚0 disk.sda2.out‚0) disk./.usagePercent‚42.95015361271278 disk.sda3.out‚ 380.30859375 proc.cpu.threads‚17 network.recv‚ 2.204843e+06 disk.drbd2165.out‚0# disk.dm-0.out‚106211.5634765625& disk.nvme0n1.out‚106211.5634765625 disk.sda1.out‚0 disk.rbd0.in‚0‹ulŸ¥‚(:” űëÒ³•¸Ç gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lˆø如(:Ú űëÒŠÉ×É network.recv‚ 2.208421e+06 disk.rbd1.in‚0 disk.sda3.out‚ 380.62109375 proc.cpu.threads‚17) disk./.usagePercent‚43.08899545879499 disk.dm-1.out‚ 380.62109375 disk.sda.out‚ 380.62109375 disk.nvme0c0n1.in‚ 25234.4375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚1.96875$ memory_percent‚88.12038898468018 network.sent‚ 9.672884e+06 disk.nvme0n1.in‚ 25234.4375 disk.rbd0.out‚0# proc.memory.rssMB‚ 2942.61328125* proc.memory.percent‚17.960286140441895 cpu‚25.548759661927793 disk.sda2.in‚0) disk.nvme0c0n1.out‚108466.42333984375' disk.nvme0n1.out‚108466.42333984375$ disk./.usageGB‚642.0571708679199 disk.rbd1.out‚ 23.7021484375 disk.sda.in‚1.96875 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.dm-1.in‚1.96875 disk.drbd2165.in‚0$ disk.dm-0.out‚108466.42333984375 disk.rbd0.in‚0 disk.dm-0.in‚ 25234.4375) proc.memory.availableMB‚ 1946.35546875«Ë®¿ž§‚(:“ Ô±ëÒŒÀ¦Ç gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚268.527' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.70539999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c`iS(:ã Ô±ëÒ‹°ùÈ disk.sda2.out‚0 disk.drbd2165.out‚0" memory_percent‚88.120436668396 cpu‚25.549049338100936) disk.nvme0c0n1.out‚108757.95947265625' disk.nvme0n1.out‚108757.95947265625 disk.rbd0.in‚0 disk.rbd1.out‚ 23.7021484375 disk.drbd2165.in‚0 disk.dm-0.in‚25235.26953125$ disk.dm-0.out‚108757.95947265625# proc.memory.rssMB‚ 2942.61328125 network.recv‚ 2.211668e+06 disk.dm-1.in‚1.96875 disk.rbd1.in‚0 disk.sda3.in‚1.96875* proc.memory.percent‚17.960286140441895$ disk./.usageGB‚644.0236854553223 disk.dm-1.out‚ 380.91015625 disk.sda1.out‚0 network.sent‚ 9.67775e+06 disk.sda3.out‚ 380.91015625 proc.cpu.threads‚17) proc.memory.availableMB‚ 1946.34765625 disk.sda2.in‚0 disk.sda.in‚1.96875$ disk.nvme0c0n1.in‚25235.26953125 disk.sda1.in‚0) disk./.usagePercent‚43.22096990277121 disk.rbd0.out‚0 disk.sda.out‚ 380.91015625" disk.nvme0n1.in‚25235.26953125:NeÂ2©- _wandbruntime‚8816  _runtime‚8816dÒ§bǪÁ & lr-AdamW/pg2‚0.0004185396356531674 trainer/global_step‚5049 % _timestamp‚1.7843386558237484e+09 _runtime‚8816.408830143 _step‚412 & lr-AdamW/pg1‚0.0004185396356531674dcÒ¸Ì«Æ & lr-AdamW/pg1‚0.0004185396356531674 & lr-AdamW/pg2‚0.0004185396356531674 trainer/global_step‚5049 % _timestamp‚1.7843386558237484e+09 _runtime‚8816.408830143 _step‚412œ¦×Î2¬- _wandbruntime‚8816  _runtime‚8816ã ë­÷O JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43028743 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1449926940352512 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1449926940352512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _runtime‚ 8816.41005208 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4748660866048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1454674255828992 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4740537381011 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4739192780689 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1449926966525952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43028743 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1449300375384064 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86733798 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1454675627392000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_poolE´ é.allocated‚43715034 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14334110 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12953370 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4748660866048 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1454675627392000 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43028737 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12953365 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚392075825197056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 $ _timestamp‚1.784338655824832e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43715034 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43028737 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1454674255828992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27287730 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1454039568164753ÞX;Š(®üO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4747315476480 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12953365 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚392075825197056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 $ _timestamp‚1.784338655824832e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43715034 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43028737 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1454674255828992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27287730 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1454039568164753 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43028743 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1449926940352512 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1449926940352512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _runtime‚ 8816.41005208 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4748660866048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1454674255828992 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4740537381011 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4739192780689 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1449926966525952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43028743 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1449300375384064 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86733798 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1454675627392000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43715034 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14334110 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12953370 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4748660866048 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1454675627392000 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43028737 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1454953181594624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1454318373409175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14336846 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚398272344447488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚392153311604736 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4740116246935 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43713539 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86760475 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚5049 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4749585908736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4749585908736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14337083 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27292754 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚392152943513600 _step‚414h —e·±ˆN REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12955908 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1454954553164800 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1450204967256064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4749585908736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14337083 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27292754 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚392152943513600 _step‚414 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1450204967256064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43723526 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6118998313472 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1454953181594624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocateÞR W d‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1454318373409175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14336846 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚398272344447488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚392153311604736 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4740116246935 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43713539 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86760475 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚5049 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4749585908736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1468620608265216 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1463827003031552 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87579829 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13080520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1463194464292864 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4785429893797 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚402127361540608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4793631407104 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚5099 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44140786 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1463826976858112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1467979894186661 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44130813 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4793631407104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1463827003031552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44140786 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4786774494119 _step‚417 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44130813 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14471956 % _timestamp‚1.7843387361478488e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4794976796672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27552471 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1468621979828224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43439037 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1468620608265216 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚395950145581056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1463194464292864 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4785429893797 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚402127361540608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4793631407104 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚5099 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44140786 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1463826976858112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1467979894186661 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44130813 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4793631407104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1463827003031552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44140786 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4786774494119 _step‚417 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44130813 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14471956 % _timestamp‚1.7843387361478488e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4794976796672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27552471 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1468621979828224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43439037 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1468620608265216 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚395950145581056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1468620608265216 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1463827003031552 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87579829 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13080520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27557725 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚396027263897600 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13083058 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6178775153152 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1464105003761664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44149278 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43447243 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚418 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1464105003761664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1468900905601024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4795901839360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚8898.130573741 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13083063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14474667 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4787697960477 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43447249 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5099 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44139291 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87586534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4795901839360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4786353360043 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚402206039050752 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44149278 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4794556442624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1468899534030848 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87596527 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1468260069459485 $ _timestamp‚1.784338737545324e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4794556442624 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87596527 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44139291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1468258699431083 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1463472371499008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43447243 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14474911 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1463472346071040 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27557974 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87586534 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43447249 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87596527 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1468260069459485 $ _timestamp‚1.784338737545324e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4794556442624 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87596527 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44139291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1468258699431083 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1463472371499008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43447243 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14474911 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1463472346071040 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27557974 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87586534 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43447249 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27557725 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚396027263897600 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13083058 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6178775153152 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1464105003761664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44149278 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43447243 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚418 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1464105003761664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1468900905601024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4795901839360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚8898.130573741 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13083063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14474667 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4787697960477 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43447249 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5099 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44139291 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87586534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4795901839360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4786353360043 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚402206039050752 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44149278 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4794556442624¢Üõ~2Ê- _wandbruntime‚8898  _runtime‚8898“©ìË‚ _runtime‚8898.210105122 %loss/total_loss‚8.551974151524256 0loss/FRt1/assignment_loss‚1.6617121189042474 /loss/FRt1/detection_loss‚0.5759137806402866 /loss/FBt1/detection_loss‚0.7169086322614331 /loss/FBt2/detection_loss‚0.8825228754431009 0loss/FRt2/assignment_loss‚0.7602720730802388 2loss/SRqqt2/assignment_loss‚0.5011448055388311 1loss/FBt1/assignment_loss‚0.02232087975311064 epoch‚12 _step‚419 2loss/SRqqt1/assignment_loss‚0.4674887726578933 3loss/FBt2/assignment_loss‚0.0022079808604829054 /loss/FRt2/detection_loss‚1.5427998810174686 1loss/SRqqt1/detection_loss‚0.7446062012098819 % _timestamp‚1.7843387376253326e+09 /loss/SRqqt2/detection_loss‚0.67407615015728 trainer/global_step‚5099£]¥% ̇ /loss/SRqqt2/detection_loss‚0.67407615015728 trainer/global_step‚5099 _runtime‚8898.210105122 %loss/total_loss‚8.551974151524256 0loss/FRt1/assignment_loss‚1.6617121189042474 /loss/FRt1/detection_loss‚0.5759137806402866 /loss/FBt1/detection_loss‚0.7169086322614331 /loss/FBt2/detection_loss‚0.8825228754431009 0loss/FRt2/assignment_loss‚0.7602720730802388 2loss/SRqqt2/assignment_loss‚0.5011448055388311 1loss/FBt1/assignment_loss‚0.02232087975311064 epoch‚12 _step‚419 2loss/SRqqt1/assignment_loss‚0.4674887726578933 3loss/FBt2/assignment_loss‚0.0022079808604829054 /loss/FRt2/detection_loss‚1.5427998810174686 1loss/SRqqt1/detection_loss‚0.7446062012098819 % _timestamp‚1.7843387376253326e+09£sÏ ’Í‚(:‡ ½²ëÒâóÇ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚206.05' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.21 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯x=¼Ö΂(:Ë ½²ëÒý™žÉ! disk.nvme0n1.in‚ 25452.7109375$ memory_percent‚88.12665939331055 network.sent‚ 9.770886e+06 disk.sda2.in‚0! disk.dm-0.out‚114734.64453125# proc.memory.rssMB‚ 2942.61328125 network.recv‚ 2.237374e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚114734.64453125$ disk.nvme0n1.out‚114734.64453125 disk.rbd0.out‚0* proc.memory.percent‚17.960286140441895# disk./.usageGB‚617.126392364502 disk.dm-0.in‚ 25452.7109375 disk.sda1.out‚0 cpu‚25.551129244681245 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda3.out‚386.0625 disk.rbd0.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 1945.328125 disk.dm-1.out‚386.0625 disk.sda.in‚1.96875 disk.sda.out‚386.0625# disk.nvme0c0n1.in‚ 25452.7109375) disk./.usagePercent‚41.41586999511414 disk.dm-1.in‚1.96875 disk.rbd1.out‚ 24.0615234375 disk.sda1.in‚0 disk.sda3.in‚1.96875-éLєς(:‰ ̲ëÒá”ÛÈ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚265.623' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ¥´)ÝЂ(:Ò ̲ëÒ™ªÁÉ$ disk./.usageGB‚619.1697959899902) disk.nvme0c0n1.out‚116885.08056640625 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.out‚386.375 disk.sda.in‚1.96875# proc.memory.rssMB‚ 2942.61328125 proc.cpu.threads‚17 network.recv‚ 2.241913e+06( disk./.usagePercent‚41.5530045269505 disk.drbd2165.in‚0 cpu‚25.55132125381022 disk.drbd2165.out‚0' proc.memory.availableMB‚ 1945.078125' disk.nvme0n1.out‚116885.08056640625 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0* proc.memory.percent‚17.960286140441895$ disk.nvme0c0n1.in‚25453.67578125" disk.nvme0n1.in‚25453.67578125 disk.sda1.in‚0 disk.sda3.out‚386.375 disk.rbd0.in‚0 disk.rbd1.out‚ 24.0615234375# memory_percent‚88.1281852722168 disk.sda3.in‚1.96875 disk.dm-1.in‚1.96875 disk.sda.out‚386.375 network.sent‚ 9.833641e+06 disk.dm-0.in‚25453.67578125$ disk.dm-0.out‚116885.08056640625â‘£ žÑ‚(:“ Û²ëÒå‰É gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚327.145' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.42899999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±>ÿáÒ‚(:Ö Û²ëÒ¸¤ˆË disk.sda.in‚1.96875$ memory_percent‚88.12668323516846 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚119224.39453125 disk.dm-1.out‚ 386.734375! disk.dm-0.out‚119224.39453125) proc.memory.availableMB‚ 1945.32421875" disk.nvme0n1.in‚25453.67578125 disk.sda3.in‚1.96875 disk.rbd0.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2942.61328125) disk./.usagePercent‚41.68847859391979 disk.sda3.out‚ 386.734375 disk.sda.out‚ 386.734375 disk.dm-0.in‚25453.67578125 disk.dm-1.in‚1.96875 network.recv‚ 2.245418e+06$ disk./.usageGB‚621.1884574890137 disk.sda1.in‚0 disk.rbd1.in‚0& disk.nvme0c0n1.out‚119224.39453125 cpu‚25.551672299801407$ disk.nvme0c0n1.in‚25453.67578125 network.sent‚ 9.84067e+06 disk.sda1.out‚0 disk.rbd1.out‚ 24.2021484375 disk.drbd2165.in‚0* proc.memory.percent‚17.960286140441895 proc.cpu.threads‚17½äÐŽŸÓ‚(:” ê²ëÒŸòÕÈ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚286.023' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.204600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G.ú¹ïÔ‚(:ä ê²ëүɀÉ) disk.nvme0c0n1.out‚120773.02978515625 disk.rbd1.out‚ 24.2021484375 disk.drbd2165.in‚0" disk.nvme0n1.in‚25554.72265625 disk.dm-1.in‚1.96875) proc.memory.availableMB‚ 1945.32421875 disk.dm-0.in‚25554.72265625 network.sent‚ 9.843698e+06 network.recv‚ 2.248934e+06) disk./.usagePercent‚41.82501278808835 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 387.10546875* proc.memory.percent‚17.960286140441895$ disk./.usageGB‚623.2229156494141$ disk.nvme0c0n1.in‚25554.72265625 disk.sda1.out‚0 disk.sda2.in‚0 cpu‚25.5519044845648' disk.nvme0n1.out‚120773.02978515625 disk.sda3.in‚1.96875 disk.sda2.out‚0# proc.memory.rssMB‚ 2942.61328125$ disk.dm-0.out‚120773.02978515625 disk.drbd2165.out‚0 disk.rbd0.in‚0$ memory_percent‚88.12668323516846 disk.sda.in‚1.96875 disk.sda3.out‚ 387.10546875 disk.dm-1.out‚ 387.10546875 proc.cpu.threads‚17˜lÆ^ŸÕ‚(:” ù²ëÒåÓÈ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚274.182' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.836400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®A'æÖ‚(:Û ù²ëÒñ²É) disk.nvme0c0n1.out‚123111.99267578125 disk.sda2.out‚0$ memory_percent‚88.12668323516846* disk./.usagePercent‚41.961702122822665 disk.rbd1.out‚ 24.2646484375 disk.sda.in‚1.96875* proc.memory.percent‚17.960286140441895 disk.nvme0n1.in‚ 25555.6875# proc.memory.rssMB‚ 2942.61328125 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 387.35546875) proc.memory.availableMB‚ 1945.32421875' disk.nvme0n1.out‚123111.99267578125 network.recv‚ 2.252582e+06 disk.nvme0c0n1.in‚ 25555.6875$ disk.dm-0.out‚123111.99267578125 disk.sda3.in‚1.96875 disk.rbd1.in‚0$ disk./.usageGB‚625.2596855163574 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 387.35546875 cpu‚25.552122295717957 proc.cpu.threads‚17 network.sent‚ 9.848813e+06 disk.drbd2165.in‚0 disk.dm-0.in‚ 25555.6875 disk.rbd0.out‚0 disk.dm-1.in‚1.96875 disk.sda.out‚ 387.35546875Šxà=2×- _wandbruntime‚8976  _runtime‚8976Åç‰ÖÈØ ' lr-AdamW/pg1‚0.00041589500560757663 ' lr-AdamW/pg2‚0.00041589500560757663 trainer/global_step‚5149 % _timestamp‚1.7843388160748713e+09 _runtime‚ 8976.66106215 _step‚420v©þÍÙÇ ' lr-AdamW/pg1‚0.00041589500560757663 ' lr-AdamW/pg2‚0.00041589500560757663 trainer/global_step‚5149 % _timestamp‚1.7843388160748713e+09 _runtime‚ 8976.66106215 _step‚420¤Œ¬C2Ú- _wandbruntime‚8976  _runtime‚8976âbÏóJ ÛùO WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1482568332264448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27817704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6237781646848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43849337 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14610274 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1477727039537152 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1482566960701440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all:‰µ.allocated‚ 49071259648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4831667006905 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1482568332264448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1477727039537152 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚399824097873920 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚421 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44566538 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4839947337728 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1477088553201664 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44556565 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1477727013363712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚399824465965056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1482566960701440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43849343 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0À&æ*(ÜþO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1481920220208569 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843388160785263e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43849337 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1477088553201664 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44556565 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1477727013363712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚399824465965056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1482566960701440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43849343 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1482568332264448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27817704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6237781646848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43849337 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14610274 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1477727039537152 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1482566960701440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4831667006905 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1482568332264448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1477727039537152 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚399824097873920 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚421 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44566538 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4839947337728 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399901584281600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚422 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6238948739584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88422586 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27822988 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13210213 % _timestamp‚1.7843388174787102e+09 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚8978.063946379 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43857543 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4840872373248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1478005014093824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88432579 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44575030 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43857549 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4842217769984 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44565043 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44575030 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4840872373248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1477366460407808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚406140533021184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4833935073585 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6238981171712 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1482199025452991 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1477366434979840 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43857543 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4832590473151 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88422586 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1482200395481393 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚5149 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚406140933544448 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1482847258037248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4842217769984 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175"HÀÉé߈N UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4840872373248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1477366460407808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚406140533021184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4833935073585 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6238981171712 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1482199025452991 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1477366434979840 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43857543 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4832590473151 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88422586 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1482200395481393 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚5149 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚406140933544448 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1482847258037248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4842217769984 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399901584281600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚422 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6238948739584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88422586 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.a&Õ ¶%ll.freed‚27822988 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13210213 % _timestamp‚1.7843388174787102e+09 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚8978.063946379 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43857543 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4840872373248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1478005014093824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88432579 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44575030 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43857549 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4842217769984 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44565043 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44575030 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¦×½{«2à- _wandbruntime‚8978  _runtime‚8978áãÿá 2loss/SRqqt1/assignment_loss‚0.5128684202500745 1loss/FBt2/assignment_loss‚0.08160203168880585 0loss/SRqqt2/detection_loss‚0.661812902061437 epoch‚12 % _timestamp‚1.7843388175615754e+09 1loss/FBt1/assignment_loss‚0.00981460457636381 /loss/FRt1/detection_loss‚0.5187396830577891 1loss/SRqqt1/detection_loss‚0.8137466881025411 /loss/FBt1/detection_loss‚0.5121268366259357 /loss/FBt2/detection_loss‚0.9237511329232685 trainer/global_step‚5149 _runtime‚8978.146369733 _step‚423 0loss/FRt2/assignment_loss‚0.6779910380323532 3loss/SRqqt2/assignment_loss‚0.47438056147769414 .loss/FRt2/detection_loss‚1.565146010940243 %loss/total_loss‚8.362947260932392 0loss/FRt1/assignment_loss‚1.6109673511958862áã‡9 ↠.loss/FRt2/detection_loss‚1.565146010940243 %loss/total_loss‚8.362947260932392 0loss/FRt1/assignment_loss‚1.6109673511958862 2loss/SRqqt1/assignment_loss‚0.5128684202500745 1loss/FBt2/assignment_loss‚0.08160203168880585 0loss/SRqqt2/detection_loss‚0.661812902061437 epoch‚12 % _timestamp‚1.7843388175615754e+09 1loss/FBt1/assignment_loss‚0.00981460457636381 /loss/FRt1/detection_loss‚0.5187396830577891 1loss/SRqqt1/detection_loss‚0.8137466881025411 /loss/FBt1/detection_loss‚0.5121268366259357 /loss/FBt2/detection_loss‚0.9237511329232685 trainer/global_step‚5149 _runtime‚8978.146369733 _step‚423 0loss/FRt2/assignment_loss‚0.6779910380323532 3loss/SRqqt2/assignment_loss‚0.47438056147769414§ –'¶‘ã‚(:† ˆ³ëÒòÞçÊ gpu.0.gpu‚22.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥7Dàä‚(:Õ ˆ³ëÒÙ‡ÏÌ disk.sda.out‚ 387.65234375 disk.dm-0.in‚ 25555.6875 disk.nvme0n1.in‚ 25555.6875 disk.rbd1.in‚0 proc.cpu.threads‚17# disk.dm-0.out‚125466.4404296875) disk./.usagePercent‚42.09722457521591$ disk./.usageGB‚627.2790679931641 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 387.65234375 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚125466.4404296875( disk.nvme0c0n1.out‚125466.4404296875# proc.memory.rssMB‚ 2942.61328125 cpu‚25.552431618875158 disk.dm-1.out‚ 387.65234375' proc.memory.availableMB‚ 1945.734375 disk.sda.in‚1.96875 disk.sda3.in‚1.96875 disk.sda2.in‚0* proc.memory.percent‚17.960286140441895 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚1.96875 network.sent‚ 9.866597e+06 disk.nvme0c0n1.in‚ 25555.6875 disk.sda2.out‚0 disk.rbd1.out‚ 24.2646484375$ memory_percent‚88.12417984008789 network.recv‚ 2.255938e+06TnZŸå‚(:” —³ëÒ¾ÕçÇ gpu.0.gpu‚89.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚286.023' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.204600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*Kɺäæ‚(:Ù —³ëÒ«ÜŒÉ disk.sda.out‚ 388.21875$ disk.nvme0c0n1.in‚25556.51953125( disk.nvme0c0n1.out‚127020.0751953125 disk.rbd0.out‚0$ memory_percent‚88.12413215637207 network.recv‚ 2.26071e+06* disk./.usagePercent‚42.232127745383835 disk.rbd1.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 1945.7421875 network.sent‚ 9.916684e+06 disk.sda.in‚1.96875# disk.dm-0.out‚127020.0751953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 24.3427734375 disk.drbd2165.out‚0& disk.nvme0n1.out‚127020.0751953125 disk.sda3.out‚ 388.21875 disk.sda2.in‚0 disk.dm-1.in‚1.96875 cpu‚25.552642313197932 disk.drbd2165.in‚0 disk.dm-1.out‚ 388.21875# proc.memory.rssMB‚ 2942.61328125 disk.dm-0.in‚25556.51953125 disk.sda3.in‚1.96875 proc.cpu.threads‚17$ disk./.usageGB‚629.2892227172852" disk.nvme0n1.in‚25556.51953125* proc.memory.percent‚17.960286140441895ŠH:“ç‚(:ˆ ¦³ëÒ‰²ÈÊ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚282.295' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.459 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à:Líè‚(:â ¦³ëÒ”ßÑÌ disk.rbd1.out‚ 24.3427734375 disk.sda.out‚ 388.7109375 network.recv‚ 2.263931e+06) disk.nvme0c0n1.out‚129741.21142578125 disk.dm-0.in‚25629.57421875 disk.sda1.in‚0$ memory_percent‚88.12413215637207$ disk./.usageGB‚632.1163330078125 disk.sda2.in‚0 disk.dm-1.out‚ 388.7109375' disk.nvme0n1.out‚129741.21142578125# proc.memory.rssMB‚ 2942.61328125* proc.memory.percent‚17.960286140441895 disk.sda.in‚1.96875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 388.7109375 network.sent‚ 9.923589e+06 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17( proc.memory.availableMB‚ 1945.7421875 disk.rbd1.in‚0$ disk.dm-0.out‚129741.21142578125 disk.sda3.in‚1.96875 disk.rbd0.in‚0) disk./.usagePercent‚42.42185748908466 disk.dm-1.in‚1.96875$ disk.nvme0c0n1.in‚25629.57421875" disk.nvme0n1.in‚25629.57421875 cpu‚25.552921250672746cðS’é‚(:‡ µ³ëÒ÷’²É gpu.0.gpu‚4.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚320.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.0694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×NÃÝåê‚(:Ú µ³ëÒ¬Ü¾Ë disk.sda2.in‚0 cpu‚25.553254452657598 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚25629.57421875" disk.nvme0n1.in‚25629.57421875$ memory_percent‚88.12408447265625# proc.memory.rssMB‚ 2942.61328125* disk./.usagePercent‚42.465942498495906 disk.dm-1.in‚1.96875 disk.sda3.out‚ 389.19140625 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda.in‚1.96875 disk.sda2.out‚0 network.recv‚ 2.26746e+06 disk.drbd2165.out‚0 disk.sda3.in‚1.96875 disk.dm-1.out‚ 389.19140625 disk.rbd1.out‚ 24.4052734375 disk.sda1.out‚0# proc.memory.availableMB‚1945.75 network.sent‚ 9.928604e+06' disk.nvme0c0n1.out‚132609.740234375 disk.rbd0.in‚0* proc.memory.percent‚17.960286140441895$ disk./.usageGB‚632.7732315063477 disk.sda.out‚ 389.19140625 disk.dm-0.in‚25629.57421875" disk.dm-0.out‚132609.740234375% disk.nvme0n1.out‚132609.740234375 disk.sda1.in‚0Ú¦ŠW“ë‚(:ˆ ijëÒ½è¿Ç gpu.0.gpu‚89.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚287.615' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.523 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔWÒDÝì‚(:Ò ijëÒ̧É disk.sda.out‚ 389.515625 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.recv‚ 2.270624e+06$ disk./.usageGB‚634.1160049438477# disk.dm-0.out‚134161.9482421875 disk.nvme0n1.in‚ 25630.40625 disk.rbd0.out‚0 disk.drbd2165.in‚0& disk.nvme0n1.out‚134161.9482421875 disk.sda2.in‚0 cpu‚25.55348244070879 disk.dm-1.out‚ 389.515625 disk.rbd1.in‚0 disk.sda.in‚1.96875) disk./.usagePercent‚42.55605715054849 disk.sda1.out‚0( proc.memory.availableMB‚ 1945.4921875 network.sent‚ 9.933568e+06 disk.rbd1.out‚ 24.4052734375* proc.memory.percent‚17.960286140441895! disk.nvme0c0n1.in‚ 25630.40625( disk.nvme0c0n1.out‚134161.9482421875 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.in‚1.96875$ memory_percent‚88.12565803527832 disk.sda1.in‚0 disk.sda3.in‚1.96875# proc.memory.rssMB‚ 2942.61328125 disk.dm-0.in‚ 25630.40625 disk.sda3.out‚ 389.5156254¸ð2í- _wandbruntime‚9056  _runtime‚9056~^Õ™Éîà ' lr-AdamW/pg1‚0.00041455582097330213 ' lr-AdamW/pg2‚0.00041455582097330213 trainer/global_step‚5199 % _timestamp‚1.7843388963678486e+09 _runtime‚9056.952908888 _step‚424»âpÎïÈ ' lr-AdamW/pg1‚0.00041455582097330213 ' lr-AdamW/pg2‚0.00041455582097330213 trainer/global_step‚5199 % _timestamp‚1.7843388963678486e+09 _runtime‚9056.952908888 _step‚424¨Ë™2ð- _wandbruntime‚9056  _runtime‚9056mVÿ'ñùO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4887608657920 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44992290 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403698786349056 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44259637 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1491627049869312 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89251933 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14748196 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚409996823683584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89241954 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6298037334528 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _runtime‚9056.957859196 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4886263268352 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1495860546230477 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1490982667538432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1496513313137664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1496514684700672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14748444 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚409996423153152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1496514684700672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4879248720335 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28083011 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1495861916258767 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6298004895232 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4877904120013 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6298037334528 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _runtime‚9056.957859196 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4886263268352 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1495860546230477 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1490982667538432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1496513313137664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1496514684700672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14748444 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚409996423153152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1496514684700672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4879248720335 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28083011 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1495861916258767 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6298004895232 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4877904120013 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44259637 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1491627049869312©ßÚ-2ó- _wandbruntime‚9058  _runtime‚9058-?Q'ô‚N HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1496139351474899 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44267849 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1491260549316608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89258638 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13337363 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13337358 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4880172186693 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4887188303872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1491905076772864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1496792238903296 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89268631 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1491905076772864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1491905050599424 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44267843 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44990795 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6299237547520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 % _timestamp‚1.7843388977682045e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 trainer/global_step‚5199 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89268631 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1491905050599424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14750920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚410075510304256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚426 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4887188303872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44990795 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89258638 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44267843 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28088278 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403776272756736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚410075109780992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1496793610473472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14751156 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45000782 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28088519 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6299205115392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4878827586259 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1491260523888640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4888533700608 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1496793610473472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚ 9058.35349601 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4888533700608 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45000782 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403775904665600Zéçº 'õ‡N NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44990795 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89258638 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44267843 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28088278 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403776272756736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚410075109780992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1496793610473472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14751156 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45000782 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28088519 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6299205115392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4878827586259 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1491260523888640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4888533700608 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1496793610473472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚ 9058.35349601 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4888533700608 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45000782 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403775904665600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1496139351474899 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44267849 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1491260549316608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89258638 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13337363 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13337358 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4880172186693 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4887188303872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1491905076772864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1496792238903296 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89268631 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1491905076772864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1491905050599424 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44267843 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44990795 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6299237547520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 % _timestamp‚1.7843388977682045e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 trainer/global_step‚5199 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89268631 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1491905050599424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14750920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚410075510304256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚426 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4887188303872ªCqò2ö- _wandbruntime‚9058  _runtime‚9058㯋O÷‚ 2loss/FBt1/assignment_loss‚0.039678543227478366 1loss/SRqqt1/detection_loss‚0.7453065434260269 trainer/global_step‚5199 1loss/FBt2/assignment_loss‚0.02355013490045535 %loss/total_loss‚8.386485466712973 0loss/FRt1/assignment_loss‚1.6898838196400259 0loss/FRt2/assignment_loss‚0.7901734568882938 2loss/SRqqt1/assignment_loss‚0.5110160667094209 /loss/FRt2/detection_loss‚1.6052287671951326 .loss/FBt1/detection_loss‚0.632311400814272 /loss/FBt2/detection_loss‚0.6088484439644075 % _timestamp‚1.7843388978462126e+09 _runtime‚9058.430994245 2loss/SRqqt2/assignment_loss‚0.4948700225832702 /loss/FRt1/detection_loss‚0.5554539035944874 1loss/SRqqt2/detection_loss‚0.6901643637697027 epoch‚12 _step‚427.°- ø‡ /loss/FBt2/detection_loss‚0.6088484439644075 % _timestamp‚1.7843388978462126e+09 _runtime‚9058.430994245 2loss/SRqqt2/assignment_loss‚0.4948700225832702 /loss/FRt1/detection_loss‚0.5554539035944874 1loss/SRqqt2/detection_loss‚0.6901643637697027 epoch‚12 _step‚427 2loss/FBt1/assignment_loss‚0.039678543227478366 1loss/SRqqt1/detection_loss‚0.7453065434260269 trainer/global_step‚5199 1loss/FBt2/assignment_loss‚0.02355013490045535 %loss/total_loss‚8.386485466712973 0loss/FRt1/assignment_loss‚1.6898838196400259 0loss/FRt2/assignment_loss‚0.7901734568882938 2loss/SRqqt1/assignment_loss‚0.5110160667094209 /loss/FRt2/detection_loss‚1.6052287671951326 .loss/FBt1/detection_loss‚0.632311400814272«ÝÛ²éù‚(:… Ó³ëÒÓǨÉ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿=³çú‚(:Ü Ó³ëÒТË disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 25630.40625$ disk.dm-0.out‚135061.59228515625# proc.memory.rssMB‚ 2942.61328125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.in‚1.96875 disk.sda3.in‚1.96875 disk.rbd0.out‚0$ memory_percent‚88.12410831451416# disk./.usageGB‚636.244987487793 disk.rbd1.out‚ 24.5302734375 disk.dm-0.in‚ 25630.40625 network.recv‚ 2.273424e+06 disk.sda.in‚1.96875 disk.sda3.out‚ 389.88671875 network.sent‚ 9.935573e+06 disk.sda2.out‚0' disk.nvme0n1.out‚135061.59228515625* proc.memory.percent‚17.960286140441895) disk./.usagePercent‚42.69893495540799 disk.sda.out‚ 389.88671875 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚135061.59228515625 disk.nvme0n1.in‚ 25630.40625 cpu‚25.554208371474683 disk.dm-1.out‚ 389.88671875 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1945.7460937583ò”û‚(:‰ â³ëÒÔ¯ŽÉ gpu.0.gpu‚98.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚337.816' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.5632 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e»¸¢Òü‚(:Ç â³ëÒ̸üÊ disk.rbd1.out‚ 24.5302734375 disk.sda.out‚ 390.53515625 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 25690.4921875* proc.memory.percent‚17.960286140441895( disk./.usagePercent‚42.8397140105018$ disk./.usageGB‚638.3426971435547! disk.nvme0n1.in‚ 25690.4921875 disk.sda3.in‚1.96875 disk.sda3.out‚ 390.53515625 disk.sda1.out‚0 disk.dm-1.in‚1.96875 disk.sda.in‚1.96875 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2942.61328125 disk.sda2.in‚0 disk.dm-1.out‚ 390.53515625 disk.dm-0.out‚ 137375.25$ memory_percent‚88.12382221221924) proc.memory.availableMB‚ 1945.79296875 disk.sda1.in‚0 disk.dm-0.in‚ 25690.4921875 disk.sda2.out‚0 disk.drbd2165.in‚0 cpu‚25.558268779174256 proc.cpu.threads‚17 network.sent‚ 1.000049e+07 disk.nvme0n1.out‚ 137375.25 disk.nvme0c0n1.out‚ 137375.25 network.recv‚ 2.277844e+06 disk.rbd1.in‚0'ˆÌ2ý- _wandbruntime‚9085  _runtime‚9085E9[`¹þ„{ =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1458333432674408 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6144294738769531 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4512968063354492 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13243506848812103 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8520786762237549 particle/accuracy_1_of_1‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6680029630661011 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9185822010040283 particle/accuracy_2_of_3‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6196653842926025 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23414747416973114 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48911330103874207 epoch‚12 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2643881142139435 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.949999988079071 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.15448659658432007 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5780812501907349 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6051863431930542 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6337341666221619 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21611836552619934 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2236422598361969 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_1_of_3‚0.9341996312141418 /particle/accuracy_4_of_4‚0.9547014832496643 particle/accuracy_3_of_5‚1 _runtime‚9085.578153384 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7052950859069824 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4296821057796478 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3016849160194397 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6577683687210083 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2517496645450592 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1815841794013977 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_5‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.047457098960876465 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5768237709999084 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23290488123893738 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6669496893882751 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.765164852142334 particle/accuracy_3_of_3‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.032477352768182755 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23546679317951202 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1718352884054184 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7308114767074585 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22864916920661926 jet/accuracy_2_of_5‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6852667331695557 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 'particle/f_score‚0.6443623304367065 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5727177858352661 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.610528826713562 particle/accuracy_1_of_3‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666567325592 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.834882915019989 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.765164852142334 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 trainer/global_step‚5›LÅÑ&199 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9991950988769531 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 +jet/accuracy_3_of_3‚0.07248137891292572 _step‚428 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9328646063804626 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5360357761383057 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2236422598361969 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.669707715511322 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6722627878189087 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2343660145998001 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6116263270378113 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2260783314704895 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1287996619939804 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7258986234664917 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24593079090118408 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.32999947667121887 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12092815339565277 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8284202814102173 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8586403131484985 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.3557282090187073 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 particle/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5697617530822754 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.71458899974823 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7627362012863159 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 jet/accuracy_5_of_5‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12914402782917023 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23060452938079834 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.3198413848876953 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960266351699829 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.222105473279953 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6631777882575989 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6061294674873352 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9753648638725281 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.34558457136154175 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23846136033535004 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23588943481445312 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.062394436448812485 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12092815339565277 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8488381505012512 +particle/sensitivity‚0.6377182602882385 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.042897529900074005 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34436219930648804 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5473488569259644 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.986282467842102 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7535556554794312 *jet/accuracy_1_of_4‚0.9589710235595703 +particle/specificity‚0.9065780639648438 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9956378936767578 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930034875869751 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_2_of_2‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8343348503112793 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.701068639755249 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20841798186302185 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9964625239372253 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22258131206035614 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5714612007141113 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 (jet/accuracy_2_of_3‚0.55788654088974 *jet/accuracy_3_of_4‚0.0841628834605217 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6783774495124817 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6249172687530518 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6768990755081177 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt1SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6241040229797363 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.989759087562561 % _timestamp‚1.7843389249924443e+09 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8834238648414612 *jet/accuracy_1_of_2‚0.7930641174316406 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.28500881791114807 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49252691864967346 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.15741285681724548 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9876824617385864 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8402872681617737 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941800236701965 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2393607795238495 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34052881598472595 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5668140649795532 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 +jet/accuracy_2_of_2‚0.14041045308113098 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 *jet/accuracy_2_of_4‚0.8050702214241028 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6317185163497925 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2210330218076706 *jet/accuracy_1_of_1‚0.3076534569263458 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5697617530822754 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.024710578843951225 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458134651184 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7162771821022034 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_5_of_5‚1 )loss/val_total_loss‚8.143864920004427 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9950742125511169 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.994797945022583 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13163459300994873 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6759262681007385 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.35548901557922363 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.560490608215332 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5660263895988464 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9847937822341919 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.053668707609176636 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29950425028800964 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5562499761581421 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8729263544082642 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987881779670715 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6453192830085754 jet/accuracy_4_of_5‚0aÕ!Ö=ÿ‰{ 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6759262681007385 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.35548901557922363 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.560490608215332 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5660263895988464 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9847937822341919 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.053668707609176636 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29950425028800964 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5562499761581421 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8729263544082642 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987881779670715 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6453192830085754 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1458333432674408 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6144294738769531 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4512968063354492 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13243506848812103 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8520786762237549 particle/accuracy_1_of_1‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6680029630661011 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9185822010040283 particle/accuracy_2_of_3‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6196653842926025 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23414747416973114 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48911330103874207 epoch‚12 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2643881142139435 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.949999988079071 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.15448659658432007 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5780812501907349 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6051863431930542 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6337341666221619 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21611836552619934 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2236422598361969 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_1_of_3‚0.9341996312141418 /particle/accuracy_4_of_4‚0.9547014832496643 particle/accuracy_3_of_5‚1 _runtime‚9085.578153384 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7052950859069824 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4296821057796478 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3016849160194397 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6577683687210083 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2517496645450592 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1815841794013977 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_5‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.047457098960876465 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5768237709999084 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23290488123893738 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6669496893882751 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.765164852142334 particle/accuracy_3_of_3‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.032477352768182755 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23546679317951202 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1718352884054184 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7308114767074585 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22864916920661926 jet/accuracy_2_of_5‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6852667331695557 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 'particle/f_score‚0.6443623304367065 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5727177858352661 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.610528826713562 particle/accuracy_1_of_3‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666567325592 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.834882915019989 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.765164852142334 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 trainer/global_step‚5199 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9991950988769531 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 +jet/accuracy_3_of_3‚0.07248137891292572 _step‚428 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9328646063804626 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5360357761383057 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2236422598361969 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.669707715511322 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6722627878189087 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2343660145998001 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6116263270378113 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2260783314704895 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1287996619939804 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7258986234664917 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24593079090118408 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.32999947667121887 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12092815339565277 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8284202814102173 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8586403131484985 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.3557282090187073 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 particle/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5697617530822754 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.71458899974823 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7627362012863159 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 jet/accuracy_5_of_5‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12914402782917023 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23060452938079834 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.3198413848876953 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960266351699829 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.222105473279953 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6631777882575989 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6061294674873352 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9753648638725281 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.34558457136154175 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23846136033535004 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23588943481445312 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.062394436448812485 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12092815339565277 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8488381505012512 +particle/sensitivity‚0.6377182602882385 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.042897529900074005 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34436219930648804 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5473488569259644 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.986282467842102 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7535556554794312 *jet/accuracy_1_of_4‚0.9589710235595703 +particle/specificity‚0.9065780639648438 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9956378936767578 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930034875869751 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_2_of_2‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8343348503112793 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.701068639755249 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20841798186302185 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9964625239372253 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22258131206035614 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5714612007141113 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 (jet/accuracy_2_of_3‚0.55788654088974 *jet/accuracy_3_of_4‚0.0841628834605217 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6783774495124817 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6249172687530518 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6768990755081177 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt1SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6241040229797363 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.989759087562561 % _timestamp‚1.7843389249924443e+09 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8834238648414612 *jet/accuracy_1_of_2‚0.7930641174316406 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.28500881791114807 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49252691864967346 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.15741285681724548 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9876824617385864 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8402872681617737 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941800236701965 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2393607795238495 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34052881598472595 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5668140649795532 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 +jet/accuracy_2_of_2‚0.14041045308113098 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 *jet/accuracy_2_of_4‚0.8050702214241028 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6317185163497925 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2210330218076706 *jet/accuracy_1_of_1‚0.3076534569263458 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5697617530822754 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.024710578843951225 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458134651184 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7162771821022034 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_5_of_5‚1 )loss/val_total_loss‚8.143864920004427 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9950742125511169 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.994797945022583 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13163459300994873¬ ï➀‚(:“ ñ³ëÒ•ð¯È gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚272.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.565999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ,Lé‚(:Þ ñ³ëҜ܆Ê$ disk.nvme0c0n1.in‚25691.32421875 disk.sda1.out‚0# proc.memory.rssMB‚ 2947.58984375 disk.dm-1.in‚1.96875 disk.drbd2165.in‚0" disk.nvme0n1.in‚25691.32421875 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 390.9296875 disk.drbd2165.out‚0 disk.dm-0.in‚25691.32421875 disk.sda3.in‚1.96875 disk.rbd0.in‚0$ memory_percent‚88.15732002258301 disk.sda3.out‚ 390.9296875& disk.nvme0n1.out‚139661.5322265625 disk.sda1.in‚0 network.sent‚ 1.0003412e+07 disk.rbd1.in‚0 disk.sda.in‚1.96875( disk.nvme0c0n1.out‚139660.2822265625# disk.dm-0.out‚139657.4384765625( proc.memory.availableMB‚ 1940.3046875 cpu‚25.56179858879138 disk.sda2.out‚0 disk.rbd1.out‚ 24.6552734375* proc.memory.percent‚17.990660667419434 proc.cpu.threads‚17 disk.sda.out‚ 390.9296875 network.recv‚ 2.281397e+06( disk./.usagePercent‚42.8439301985832$ disk./.usageGB‚638.4055213928223©ÌÚ˜“‚‚(:ˆ €´ëÒÚ‚ÄÉ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìöÇú郂(:Þ €´ëÒµüðË( disk.nvme0c0n1.out‚142055.2451171875 disk.sda2.in‚0 disk.dm-1.in‚1.96875 disk.rbd1.in‚0 disk.sda.in‚1.96875 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 24.6552734375# proc.memory.rssMB‚ 2947.58984375* proc.memory.percent‚17.990660667419434 disk.sda.out‚ 392.265625 disk.sda3.out‚ 392.265625 disk.sda2.out‚0 disk.dm-1.out‚ 392.265625 network.sent‚ 1.0060727e+07$ memory_percent‚88.15791606903076 cpu‚25.561960199797554# disk./.usageGB‚593.958812713623 disk.drbd2165.out‚0# disk.dm-0.out‚142055.2490234375 disk.sda1.out‚0 disk.rbd0.in‚0* disk./.usagePercent‚39.861074285849575 disk.dm-0.in‚25691.32421875 disk.sda1.in‚0 disk.sda3.in‚1.96875" disk.nvme0n1.in‚25691.32421875 network.recv‚ 2.285722e+06$ disk.nvme0c0n1.in‚25691.32421875& disk.nvme0n1.out‚142055.2451171875) proc.memory.availableMB‚ 1940.20703125 proc.cpu.threads‚17€<Ÿ„‚(:” ´ëÒôײÇ gpu.0.gpu‚89.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚231.641' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚46.328199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûVê…‚(:ß ´ëÒÄ¿É network.recv‚ 2.289487e+06 disk.rbd0.in‚0 disk.rbd1.out‚ 24.7177734375# disk.dm-0.out‚142076.9052734375$ memory_percent‚88.15348148345947* proc.memory.percent‚17.984557151794434 cpu‚25.562312155003404 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda.out‚ 393.6015625# disk.nvme0c0n1.in‚ 25692.2890625) proc.memory.availableMB‚ 1940.93359375$ disk./.usageGB‚593.9587478637695& disk.nvme0n1.out‚142076.9052734375 disk.rbd1.in‚0 disk.dm-0.in‚ 25692.2890625 disk.sda3.in‚1.96875 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 25692.2890625 disk.sda1.in‚0 disk.dm-1.out‚ 393.6015625 disk.sda.in‚1.96875# proc.memory.rssMB‚ 2946.58984375 network.sent‚ 1.0065708e+07 disk.rbd0.out‚0 disk.dm-1.in‚1.96875( disk.nvme0c0n1.out‚142076.9052734375 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd2165.out‚0* disk./.usagePercent‚39.861069933721495 disk.sda3.out‚ 393.6015625q&„È”†‚(:‰ ž´ëÒÛÚÝÇ gpu.0.gpu‚97.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚266.886' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3772 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’p¦é懂(:Û ž´ëÒňÎÉ network.sent‚ 1.007072e+07* disk./.usagePercent‚39.861073261819435# disk.nvme0c0n1.in‚ 25795.0078125! disk.nvme0n1.in‚ 25795.0078125 disk.sda1.in‚0 proc.memory.rssMB‚ 2946.9375* proc.memory.percent‚17.986679077148438 disk.dm-0.in‚ 25795.0078125 disk.rbd1.in‚0 disk.sda.in‚1.96875 proc.cpu.threads‚17 disk.dm-1.out‚ 394.10546875( disk.nvme0c0n1.out‚142084.6337890625# disk.dm-0.out‚142084.6337890625 disk.sda3.out‚ 394.10546875 cpu‚25.562561966499565 disk.rbd1.out‚ 24.7177734375 disk.sda.out‚ 394.10546875 disk.drbd2165.in‚0( proc.memory.availableMB‚ 1940.9140625 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚142084.6337890625 disk.sda1.out‚0 disk.dm-1.in‚1.96875 disk.drbd2165.out‚0# disk./.usageGB‚593.958797454834 disk.rbd0.in‚0 network.recv‚ 2.292772e+06 disk.sda3.in‚1.96875$ memory_percent‚88.15360069274902–̱ǔˆ‚(:‰ ­´ëÒÆûÉÉ gpu.0.gpu‚86.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚327.969' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.5938 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ï>²‰‚(:à ­´ëÒ¸…ÔË) disk./.usagePercent‚39.86106711763861 disk.dm-1.in‚1.96875 disk.sda.out‚ 394.41796875) disk.nvme0c0n1.out‚142090.27880859375 disk.sda1.out‚0 disk.sda3.in‚1.96875) proc.memory.availableMB‚ 1940.50390625 disk.sda2.out‚0 proc.memory.rssMB‚ 2946.9375 network.recv‚ 2.296391e+06' disk.nvme0n1.out‚142090.27880859375$ memory_percent‚88.15610408782959* proc.memory.percent‚17.986679077148438 network.sent‚ 1.0077751e+07$ disk./.usageGB‚593.9587059020996 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 25795.0078125 proc.cpu.threads‚17 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 25795.0078125 cpu‚25.562883723225458 disk.dm-1.out‚ 394.41796875 disk.rbd1.in‚0 disk.sda.in‚1.96875$ disk.dm-0.out‚142090.27880859375! disk.nvme0n1.in‚ 25795.0078125 disk.sda1.in‚0 disk.rbd1.out‚ 24.78027/håý934375 disk.sda3.out‚ 394.41796875 disk.rbd0.in‚0’ d2Š- _wandbruntime‚9164  _runtime‚9164• ôýÉ‹Ã ' lr-AdamW/pg2‚0.00041320550437564047 trainer/global_step‚5249 % _timestamp‚1.7843390036748755e+09 _runtime‚9164.259883136 _step‚429 ' lr-AdamW/pg1‚0.00041320550437564047ŒxWÎŒÈ ' lr-AdamW/pg2‚0.00041320550437564047 trainer/global_step‚5249 % _timestamp‚1.7843390036748755e+09 _runtime‚9164.259883136 _step‚429 ' lr-AdamW/pg1‚0.00041320550437564047­zS`~2- _wandbruntime‚9164  _runtime‚9164½‚ÿ'ŽùO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4929970364558 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45467051 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1509355067809792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44801949 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13470888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44801949 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚413986206447104 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28392675 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45477024 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1509355093983232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14921787 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1513632610316430 _step‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13470883 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1514294861183488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚407621218979840 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4931314964880 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44801943 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44801943 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚413986606977536 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6365355558400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1513633980344720 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1508702665379840 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4939767200256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45477024 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1514293489620480 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _runtime‚9164.261159997 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1508702639951872 % _timestamp‚1.7843390036759667e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28392419 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚407620850888704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4938421810688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6365387997696 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1514294861183488 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1509355093983232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90268994 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1509355067809792iÌu"(þO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90268994 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1509355067809792 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4929970364558 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45467051 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1509355067809792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44801949 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13470888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44801949 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚413986206447104 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28392675 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45477024 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1509355093983232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14921787 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1513632610316430 _step‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13470883 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1514294861183488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚407621218979840 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4931314964880 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44801943 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44801943 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚413986606977536 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6365355558400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1513633980344720 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1508702665379840 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4939767200256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45477024 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1514293489620480 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _runtime‚9164.261159997 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1508702639951872 % _timestamp‚1.7843390036759667e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28392419 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚407620850888704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4938421810688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6365387997696 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1514294861183488 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1509355093983232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0®‡î&Û”‚(:‰ ¼´ëÒ°™ÄÈ gpu.0.gpu‚89.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚295.976' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.1952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠØÙ‘‚(:Î ¼´ëÒ°ŠÊ network.sent‚ 1.0080774e+07$ disk./.usageGB‚593.9596405029297 disk.rbd1.in‚0 disk.sda.in‚1.96875 disk.sda1.out‚0 network.recv‚ 2.299549e+06 disk.dm-1.in‚1.96875 disk.dm-1.out‚394.625 disk.rbd1.out‚ 24.7802734375 disk.sda3.in‚1.96875 disk.rbd0.in‚0 disk.sda2.out‚0$ memory_percent‚88.15836906433105 cpu‚25.563062264587884 proc.cpu.threads‚17" disk.dm-0.out‚142097.681640625( proc.memory.availableMB‚ 1940.1328125 disk.sda2.in‚0* disk./.usagePercent‚39.861129839484505 disk.sda.out‚394.625 disk.dm-0.in‚25795.97265625" disk.nvme0n1.in‚25795.97265625% disk.nvme0n1.out‚142097.681640625$ disk.nvme0c0n1.in‚25795.97265625 disk.sda3.out‚394.625 proc.memory.rssMB‚ 2946.9375' disk.nvme0c0n1.out‚142097.681640625 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda1.in‚0* proc.memory.percent‚17.9866790771484382(2O2’- _wandbruntime‚9165  _runtime‚9165V?B& '“ƒN TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90295671 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407698337296384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90285678 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6366562340352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44810155 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45485516 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1513911415560852 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45475529 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4940692242944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1509633068539904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4940692242944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45475529 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6366594772480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚431 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1514573786956288 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1514573786956288 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13473431 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14924284 % _timestamp‚1.7843390050735652e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44810149 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4939346846208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28397710 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90295671 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1509633068539904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4930893830804 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28397949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44810155 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1514572415386112 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Ä&…ˆÍ”ˆN U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1514573786956288 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1514573786956288 VIDeviceStatsMonitorÏÛÃoA&.on_train_batch_end/inactive_split.large_pool.allocated‚13473431 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14924284 % _timestamp‚1.7843390050735652e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44810149 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4939346846208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28397710 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90295671 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1509633068539904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4930893830804 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28397949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44810155 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1514572415386112 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90295671 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407698337296384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90285678 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6366562340352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44810155 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45485516 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1513911415560852 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45475529 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4940692242944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1509633068539904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4940692242944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45475529 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6366594772480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚431¯ÉéXU2•- _wandbruntime‚9165  _runtime‚9165ûc.± –… 3loss/SRqqt1/assignment_loss‚0.48621381651320017 1loss/SRqqt1/detection_loss‚0.7392436978843393 /loss/FBt1/detection_loss‚0.4580654492324573 trainer/global_step‚5249 _runtime‚9165.739970905 0loss/FRt1/assignment_loss‚1.5379682676772404 0loss/FRt2/assignment_loss‚0.8490454078027093 2loss/FBt1/assignment_loss‚0.015310140251770464 1loss/SRqqt2/detection_loss‚0.7230830982761113 /loss/FBt2/detection_loss‚0.5673898126591336 %loss/total_loss‚8.245252016637224 epoch‚13 _step‚432 1loss/FBt2/assignment_loss‚0.10200119416845499 /loss/FRt2/detection_loss‚1.7536206678188093 2loss/SRqqt2/assignment_loss‚0.5191116464799281 0loss/FRt1/detection_loss‚0.49419881787306846 % _timestamp‚1.7843390051551867e+09¥õÁÓ—Š 0loss/FRt1/detection_loss‚0.49419881787306846 % _timestamp‚1.7843390051551867e+09 3loss/SRqqt1/assignment_loss‚0.48621381651320017 1loss/SRqqt1/detection_loss‚0.7392436978843393 /loss/FBt1/detection_loss‚0.4580654492324573 trainer/global_step‚5249 _runtime‚9165.739970905 0loss/FRt1/assignment_loss‚1.5379682676772404 0loss/FRt2/assignment_loss‚0.8490454078027093 2loss/FBt1/assignment_loss‚0.015310140251770464 1loss/SRqqt2/detection_loss‚0.7230830982761113 /loss/FBt2/detection_loss‚0.5673898126591336 %loss/total_loss‚8.245252016637224 epoch‚13 _step‚432 1loss/FBt2/assignment_loss‚0.10200119416845499 /loss/FRt2/detection_loss‚1.7536206678188093 2loss/SRqqt2/assignment_loss‚0.5191116464799281°ûÍëJ”˜‚(:‰ Ë´ëÒÍÕîÇ gpu.0.gpu‚97.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚287.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.4654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ·ç™‚(:Ü Ë´ëÒº‰ÎÉ) proc.memory.availableMB‚ 1940.42578125 proc.memory.rssMB‚ 2946.9375 disk.sda3.out‚ 394.8515625* proc.memory.percent‚17.986679077148438 network.recv‚ 2.304067e+06 disk.sda3.in‚1.96875 disk.rbd1.out‚ 24.8427734375 disk.sda.out‚ 394.8515625( disk.nvme0c0n1.out‚142103.2509765625 disk.dm-0.in‚25795.97265625" disk.nvme0n1.in‚25795.97265625& disk.nvme0n1.out‚142103.2509765625 disk.sda2.in‚0# memory_percent‚88.1565809249878 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 394.8515625 disk.sda.in‚1.96875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 cpu‚25.563315306293305 network.sent‚ 1.0132841e+07$ disk.nvme0c0n1.in‚25795.97265625* disk./.usagePercent‚39.861134959635194# disk./.usageGB‚593.959716796875 disk.dm-1.in‚1.96875# disk.dm-0.out‚142103.2509765625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd1.in‚0) =Ÿš‚(:” Ú´ëÒïÌÉ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚331.737' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.34740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*:åß›‚(:Ô Ú´ëÒÅ•ÖË disk.drbd2165.in‚0 proc.memory.rssMB‚ 2946.9375 cpu‚25.563688438536566) disk./.usagePercent‚39.86113291157491 disk.dm-1.out‚ 395.0546875 network.sent‚ 1.0187455e+07 disk.sda3.in‚1.96875 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 25908.09375( disk.nvme0c0n1.out‚142111.3505859375) proc.memory.availableMB‚ 1940.46484375 disk.sda1.out‚0 disk.sda3.out‚ 395.0546875 disk.sda2.in‚0 disk.sda.in‚1.96875 disk.sda2.out‚0 disk.dm-1.in‚1.96875 disk.drbd2165.out‚0 disk.sda.out‚ 395.0546875* proc.memory.percent‚17.986679077148438 disk.sda1.in‚0 disk.rbd1.out‚ 24.8427734375# disk.dm-0.out‚142111.3505859375$ memory_percent‚88.15634250640869 proc.cpu.threads‚17 network.recv‚ 2.308388e+06$ disk./.usageGB‚593.9596862792969 disk.nvme0n1.in‚ 25908.09375& disk.nvme0n1.out‚142111.3505859375 disk.dm-0.in‚ 25908.09375 disk.rbd0.in‚0 disk.rbd0.out‚0÷¤4j”œ‚(:‰ é´ëÒþª¹Ç gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚253.989' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7978 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒIN|ë‚(:à é´ëÒÞªÉ$ disk./.usageGB‚593.9596900939941& disk.nvme0n1.out‚142122.8759765625 disk.rbd1.out‚ 24.9052734375* proc.memory.percent‚17.986679077148438$ disk.nvme0c0n1.in‚25909.05859375 disk.dm-0.in‚25909.05859375 disk.sda3.in‚1.96875 proc.memory.rssMB‚ 2946.9375 cpu‚25.563869195874343 proc.cpu.threads‚17) disk./.usagePercent‚39.86113316758245 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚142122.8759765625 disk.dm-1.out‚ 395.58203125$ memory_percent‚88.15643787384033 network.recv‚ 2.312009e+06" disk.nvme0n1.in‚25909.05859375 disk.sda3.out‚ 395.58203125 disk.rbd0.out‚0 disk.dm-1.in‚1.96875 network.sent‚ 1.0190419e+07# disk.dm-0.out‚142122.8759765625 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.sda.in‚1.96875) proc.memory.availableMB‚ 1940.44921875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 395.58203125er žž‚(:“ ø´ëÒÒ¥ÙÉ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚276.392' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.27839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*£k⟂(:× ø´ëÒ¸çüË* proc.memory.percent‚17.986679077148438 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1941.28515625$ disk./.usageGB‚593.9598999023438 disk.sda3.out‚ 395.86328125 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.dm-0.out‚142125.9140625" disk.nvme0n1.in‚25909.05859375 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚39.86114724799683# disk.nvme0n1.out‚142125.9140625 disk.dm-1.out‚ 395.86328125 disk.sda.out‚ 395.86328125% disk.nvme0c0n1.out‚142125.9140625 cpu‚25.564101726440583 disk.rbd0.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0 network.sent‚ 1.0197449e+07 disk.sda3.in‚1.96875 disk.rbd1.out‚ 24.9052734375 disk.sda.in‚1.96875 network.recv‚ 2.315178e+06 disk.sda2.out‚0$ disk.nvme0c0n1.in‚25909.05859375 proc.memory.rssMB‚ 2946.9375 disk.dm-0.in‚25909.05859375 disk.dm-1.in‚1.96875$ memory_percent‚88.15133571624756ßJJ"” ‚(:‰ ‡µëÒ®ÆÏÇ gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚336.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.2658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀŽ±á¡‚(:Ö ‡µëÒ¶ÑÂÉ$ disk./.usageGB‚593.9600791931152" disk.nvme0c0n1.in‚ 25909.890625# disk.dm-0.out‚142136.3408203125 disk.dm-1.in‚1.96875$ memory_percent‚88.15286159515381* proc.memory.percent‚17.986679077148438) disk./.usagePercent‚39.86115928035095 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 396.2734375 network.sent‚ 1.0200524e+07 disk.sda.in‚1.96875 disk.dm-1.out‚ 396.2734375 disk.rbd1.out‚ 24.9677734375 network.recv‚ 2.318806e+06 disk.rbd0.out‚0 cpu‚25.56438984876291& disk.nvme0n1.out‚142136.3408203125 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 25909.890625 disk.sda3.in‚1.96875 disk.dm-0.in‚ 25909.890625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1941.03515625 proc.memory.rssMB‚ 2946.9375 disk.sda.out‚ 396.2734375( disk.nvme0c0n1.out‚142136.3408203125 disk.sda2.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17Â1/Â2¢- _wandbruntime‚9244  _runtime‚9244P{oŸÈ£Â ' lr-AdamW/pg1‚0.00041184414254290334 ' lr-AdamW/pg2‚0.00041184414254290334 trainer/global_step‚5299 % _timestamp‚1.7843390838148935e+09 _runtime‚ 9244.39993336 _step‚433ÖÇkÍ¤Ç ' lr-AdamW/pg2‚0.00041184414254290334 trainer/global_step‚5299 % _timestamp‚1.7843390838148935e+09 _runtime‚ 9244.39993336 _step‚433 ' lr-AdamW/pg1‚0.00041184414254290334±9ðEØ2¥- _wandbruntime‚9244  _runtime‚9244¼µìyÿ'¦ùO H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1523255104315392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45892803 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚411495171272704 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1523255130488832 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91115025 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45902776 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1522596754288640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 % _timestamp‚1.7843390838160448e+09 _runtime‚9244.401277825 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91105046 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28657515 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚417920819201536 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1527574306366628 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1527572936338338 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1528241213619712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4986083130880 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45212249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45212243 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1522596728860672 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45212243 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚417920416573952 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1528239842056704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15059209 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚411495539363840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1523255104315392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4984737741312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640g §þO IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚411495171272704 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1523255130488832 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91115025 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45902776 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1522596754288640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 % _timestamp‚1.7843390838160448e+09 _runtime‚9244.401277825 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91105046 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28657515 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚417920819201536 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1527574306366628 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1527572936338338 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1528241213619712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4986083130880 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45212249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45212243 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1522596728860672 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45212243 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚417920416573952 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1528239842056704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15059209 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚411495539363840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1523255104315392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4984737741312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1523255104315392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45892803 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995²ki´2¨- _wandbruntime‚9245  _runtime‚9245m¼S '©ƒN _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1523533131218944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91121730 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45220449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91131723 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4978475544346 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45911268 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15062237 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45220455 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 _step‚435 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚417999495799296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚411573025771520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4977130943912 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28662553 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4987008173568 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1528520139392512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45901281 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚417999093178880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6426435498496 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1527853111611162 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4987008173568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1528518767822336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13600581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4985662776832 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚5299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15061977 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91131723 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1523533105045504 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45911268 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28662553 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4987008173568 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1528520139392512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45901281 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚417999093178880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6426435498496 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1527853111611162 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4987008173568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1528518767822336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13600581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4985662776832 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚5299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15061977 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91131723 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1523533105045504 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45911268 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91131723 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4978475544346 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0³xðþ2«- _wandbruntime‚9245  _runtime‚9245Ñó­ ¬… 2loss/SRqqt2/assignment_loss‚0.5702016012769792 /loss/FRt1/detection_loss‚0.5261203092881672 trainer/global_step‚5299 % _timestamp‚1.7843390852971807e+09 0loss/FRt2/assignment_loss‚0.8613566689388849 1loss/SRqqt1/detection_loss‚0.7287412718776861 1loss/SRqqt2/detection_loss‚0.7586988732615363 /loss/FRt2/detection_loss‚1.6345503147078015 %loss/total_loss‚8.315654798613224 epoch‚13 1loss/FBt1/assignment_loss‚0.04337054823104036 2loss/FBt2/assignment_loss‚0.007739425501464073 0loss/FBt1/detection_loss‚0.42874640458231345 /loss/FBt2/detection_loss‚0.6911317513457367 _runtime‚9245.881948386 _step‚436 0loss/FRt1/assignment_loss‚1.5670043906047397 3loss/SRqqt1/assignment_loss‚0.49799323899687437mJV­Š 0loss/FRt2/assignment_loss‚0.8613566689388849 1loss/SRqqt1/detection_loss‚0.7287412718776861 1loss/SRqqt2/detection_loss‚0.7586988732615363 /loss/FRt2/detection_loss‚1.6345503147078015 %loss/total_loss‚8.315654798613224 epoch‚13 1loss/FBt1/assignment_loss‚0.04337054823104036 2loss/FBt2/assignment_loss‚0.007739425501464073 0loss/FBt1/detection_loss‚0.42874640458231345 /loss/FBt2/detection_loss‚0.6911317513457367 _runtime‚9245.881948386 _step‚436 0loss/FRt1/assignment_loss‚1.5670043906047397 3loss/SRqqt1/assignment_loss‚0.49799323899687437 2loss/SRqqt2/assignment_loss‚0.5702016012769792 /loss/FRt1/detection_loss‚0.5261203092881672 trainer/global_step‚5299 % _timestamp‚1.7843390852971807e+09´5žŒ”®‚(:‰ –µëÒšâ¬È gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚270.152' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0304 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›´ë¯‚(:à –µëÒ¾¯üÉ) disk.nvme0c0n1.out‚142139.90185546875 disk.sda3.in‚1.96875 disk.rbd1.in‚0 disk.rbd1.out‚ 24.9677734375$ memory_percent‚88.15274238586426' disk.nvme0n1.out‚142139.90185546875 disk.sda2.in‚0* proc.memory.percent‚17.986679077148438 disk.sda1.in‚0 disk.sda3.out‚ 396.53515625 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 396.53515625 network.recv‚ 2.32203e+06$ disk./.usageGB‚593.9598960876465$ disk.dm-0.out‚142139.90185546875 disk.drbd2165.in‚0( proc.memory.availableMB‚ 1941.0546875 cpu‚25.564579494661526 network.sent‚ 1.0205391e+07 disk.sda1.out‚0 proc.memory.rssMB‚ 2946.9375( disk./.usagePercent‚39.8611469919893" disk.nvme0n1.in‚26003.01953125 disk.dm-1.in‚1.96875 disk.dm-0.in‚26003.01953125 disk.sda.in‚1.96875 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚26003.01953125 disk.rbd0.in‚0 disk.sda.out‚ 396.53515625 proc.cpu.threads‚17þp:Ÿ°‚(:” ¥µëÒòâÓÉ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚302.936' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.587199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô((챂(:á ¥µëÒÇÞ…Ì proc.cpu.threads‚17 disk.sda3.in‚1.96875 disk.rbd0.out‚0 disk.sda.out‚ 398.55078125* proc.memory.percent‚17.986679077148438 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 398.55078125 disk.dm-0.in‚ 26004.6796875 disk.sda1.out‚0* disk./.usagePercent‚39.864578004963185 disk.rbd0.in‚0 disk.rbd1.out‚ 25.0302734375 network.recv‚ 2.326905e+06 disk.dm-1.in‚1.96875 disk.drbd2165.in‚0$ disk.dm-0.out‚142162.63232421875 disk.rbd1.in‚0 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚142162.63232421875$ disk./.usageGB‚594.0110206604004 disk.sda1.in‚0$ memory_percent‚88.15147876739502 network.sent‚ 1.0270201e+07 disk.sda.in‚1.96875# disk.nvme0c0n1.in‚ 26004.6796875 proc.memory.rssMB‚ 2946.9375 cpu‚25.564947491739662 disk.sda3.out‚ 398.55078125! disk.nvme0n1.in‚ 26004.6796875' disk.nvme0n1.out‚142162.63232421875) proc.memory.availableMB‚ 1941.26171875gŽøÑß²‚(:Ô ´µëÒžÁÌ disk.sda3.out‚ 399.12890625( proc.memory.availableMB‚ 1941.2578125 proc.memory.rssMB‚ 2946.9375 disk.rbd1.out‚ 25.0302734375! disk.dm-0.out‚142170.04296875 disk.sda3.in‚1.96875 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 399.12890625" disk.nvme0c0n1.in‚ 26005.515625$ disk.nvme0n1.out‚142170.04296875 disk.sda1.in‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17 network.sent‚ 1.0275231e+07 disk.sda2.out‚0 disk.dm-1.out‚ 399.12890625 disk.drbd2165.out‚0 disk.dm-0.in‚ 26005.515625 disk.rbd0.out‚0$ memory_percent‚88.15150260925293$ disk./.usageGB‚594.0110969543457 disk.nvme0n1.in‚ 26005.515625 disk.dm-1.in‚1.96875& disk.nvme0c0n1.out‚142170.04296875* proc.memory.percent‚17.986679077148438 cpu‚25.565189270296848 network.recv‚ 2.330341e+06* disk./.usagePercent‚39.864583125113874 disk.sda.in‚1.96875 disk.sda1.out‚0£V~“³‚(:ˆ ´µëÒà·ÒÌ gpu.0.gpu‚81.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚342.225' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä»rw ´‚(: $®¦‹‰ õëÒ°—€È gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚295.976' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.1952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Eí±™äµ‚(:Ù õëÒÅÍ„Ê( proc.memory.availableMB‚ 1941.0078125 disk.dm-0.in‚26005.51953125" disk.nvme0n1.in‚26005.51953125) disk./.usagePercent‚39.86457006872963 disk.sda2.out‚0 disk.sda.out‚ 399.71875$ disk./.usageGB‚594.0109024047852 disk.rbd0.out‚0 disk.dm-1.in‚1.96875) disk.nvme0c0n1.out‚142230.78271484375 proc.memory.rssMB‚ 2946.9375' disk.nvme0n1.out‚142230.78271484375 disk.sda1.out‚0$ memory_percent‚88.15302848815918 cpu‚25.565366666357317 network.recv‚ 2.334108e+06 disk.sda3.in‚1.96875 disk.rbd0.in‚0 disk.rbd1.in‚0 network.sent‚ 1.0278254e+07 disk.sda1.in‚0 disk.rbd1.out‚ 25.0927734375 disk.sda.in‚1.96875 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚26005.51953125* proc.memory.percent‚17.986679077148438 proc.cpu.threads‚17$ disk.dm-0.out‚142230.78271484375 disk.sda3.out‚ 399.71875 disk.sda2.in‚0 disk.dm-1.out‚ 399.71875ŸE …ž¶‚(:“ ÒµëÒ¢èòÇ gpu.0.gpu‚86.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚289.743' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.94859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì.Û·‚(:Ð ÒµëÒ™ìòÉ disk.sda1.out‚0 cpu‚25.56561043778114 disk.rbd0.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚142244.8017578125 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-1.out‚ 401.52734375 disk.rbd1.in‚0 disk.sda1.in‚0 proc.memory.availableMB‚1941 network.recv‚ 2.33725e+06 disk.dm-1.in‚1.96875 disk.sda.out‚ 401.52734375$ disk./.usageGB‚594.0154609680176 disk.rbd1.out‚ 25.0927734375 disk.sda.in‚1.96875 disk.drbd2165.in‚0 disk.sda3.in‚1.96875" memory_percent‚88.153076171875* proc.memory.percent‚17.986679077148438& disk.nvme0n1.out‚142244.8017578125! disk.nvme0n1.in‚ 26030.7890625 disk.sda3.out‚ 401.52734375* disk./.usagePercent‚39.864875997733044 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 26030.7890625 disk.dm-0.in‚ 26030.7890625# disk.dm-0.out‚142244.8017578125 proc.memory.rssMB‚ 2946.9375 network.sent‚ 1.028337e+07 disk.sda2.out‚0¡[š2¸- _wandbruntime‚9325  _runtime‚9325ei6ǹÁ _runtime‚9325.573507121 _step‚437 & lr-AdamW/pg1‚0.0004104718229128174 & lr-AdamW/pg2‚0.0004104718229128174 trainer/global_step‚5349 % _timestamp‚1.7843391649884455e+09"ׇBÌºÆ & lr-AdamW/pg2‚0.0004104718229128174 trainer/global_step‚5349 % _timestamp‚1.7843391649884455e+09 _runtime‚9325.573507121 _step‚437 & lr-AdamW/pg1‚0.0004104718229128174µÕÂÐ2»- _wandbruntime‚9325  _runtime‚9325rj Çÿ'¼ùO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28922985 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1537155166994432 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1542187566055936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45622549 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6484800869888 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1536490843197440 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1542186194492928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5023789191096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚9325.575012673 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚421854660617728 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45622543 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46318555 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28922708 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15197797 % _timestamp‚1.7843391649896936e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚415369859747840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46328528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1537155140820992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91941098 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1542187566055936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13725188 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1541514632388536 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45622543 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5031053671936 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175“N5…(½þO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46328528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1537155140820992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91941098 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1542187566055936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13725188 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1541514632388536 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45622543 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5031053671936 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28922985 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1537155166994432 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1542187566055936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45622549 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6484800869888 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1536490843197440 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1542186194492928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5023789191096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚9325.575012673 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚421854660617728 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45622543 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46318555 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28922708 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15197797 % _timestamp‚1.7843391649896936e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚415369859747840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097¶9É92¾- _wandbruntime‚9326  _runtime‚9326%•E~˜"¿ƒN M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5031978707456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚421932926321152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6485948256768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5033324104192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _runtime‚9326.988749063 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91967775 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1537433141551104 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 trainer/global_step‚5349 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843391664033222e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚415446978064384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421933331038720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13727726 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91967775 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1542465120258560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28928176 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46337020 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46327033 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1537433167724544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1537433167724544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5031978707456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46327033 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1542466491828736 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45630755 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1537433141551104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91957782 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15200180 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚439 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45630749 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5033324104192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45630749 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1536768724975616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚415447346155520 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28927906 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1541793437633070 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15200445 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5024712657454 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1542466491828736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46337020 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6485984883200 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ÷vó»qC6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91957782 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚03 ˨'ÀˆN NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45630755 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1537433141551104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91957782 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15200180 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚439 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45630749 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5033324104192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45630749 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1536768724975616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚415447346155520 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28927906 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1541793437633070 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15200445 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5024712657454 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1542466491828736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46337020 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6485984883200 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91957782 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5031978707456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚421932926321152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6485948256768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5033324104192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _runtime‚9326.988749063 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91967775 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1537433141551104 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 trainer/global_step‚5349 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843391664033222e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚415446978064384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421933331038720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13727726 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91967775 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1542465120258560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28928176 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46337020 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46327033 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1537433167724544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1537433167724544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5031978707456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46327033 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1542466491828736·ý3ùí2Á- _wandbruntime‚9327  _runtime‚93278!Âà „ 0loss/FRt1/assignment_loss‚1.6213244852243047 2loss/SRqqt2/assignment_loss‚0.5151395540349885 /loss/FRt1/detection_loss‚0.5099672360620664 /loss/FRt2/detection_loss‚1.6902751028401157 1loss/SRqqt2/detection_loss‚0.7091563111099775 %loss/total_loss‚8.271004641095999 epoch‚13 _step‚440 2loss/FBt2/assignment_loss‚0.005100479657288716 _runtime‚9327.061242201 0loss/FRt2/assignment_loss‚0.7830032730102539 0loss/FBt1/detection_loss‚0.41974268983914587 /loss/FBt2/detection_loss‚0.7232473688200116 % _timestamp‚1.7843391664762325e+09 2loss/SRqqt1/assignment_loss‚0.5080394017756982 1loss/FBt1/assignment_loss‚0.03943082778317264 1loss/SRqqt1/detection_loss‚0.7465779109389771 trainer/global_step‚5349<¹RÉ 1loss/FBt1/assignment_loss‚0.03943082778317264 1loss/SRqqt1/detection_loss‚0.7465779109389771 trainer/global_step‚5349 0loss/FRt1/assignment_loss‚1.6213244852243047 2loss/SRqqt2/assignment_loss‚0.5151395540349885 /loss/FRt1/detection_loss‚0.5099672360620664 /loss/FRt2/detection_loss‚1.6902751028401157 1loss/SRqqt2/detection_loss‚0.7091563111099775 %loss/total_loss‚8.271004641095999 epoch‚13 _step‚440 2loss/FBt2/assignment_loss‚0.005100479657288716 _runtime‚9327.061242201 0loss/FRt2/assignment_loss‚0.7830032730102539 0loss/FBt1/detection_loss‚0.41974268983914587 /loss/FBt2/detection_loss‚0.7232473688200116 % _timestamp‚1.7843391664762325e+09 2loss/SRqqt1/assignment_loss‚0.5080394017756982¸OÆïG”Ä‚(:‰ áµëÒ·³†È gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚288.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.6878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷±ßÅ‚(:Ô áµëÒöþâÉ* proc.memory.percent‚17.986679077148438 disk.dm-0.in‚26084.58203125$ disk./.usageGB‚594.0270156860352" disk.nvme0n1.in‚26084.58203125# disk.nvme0n1.out‚142315.9609375 disk.sda1.out‚0 disk.dm-1.out‚ 402.3359375 disk.rbd1.out‚ 25.1552734375 disk.sda.out‚ 402.3359375 disk.drbd2165.in‚0 network.sent‚ 1.0288226e+07) disk./.usagePercent‚39.86565144455426 disk.sda3.out‚ 402.3359375 disk.dm-1.in‚1.96875$ disk.nvme0c0n1.in‚26084.58203125) proc.memory.availableMB‚ 1940.94140625 proc.memory.rssMB‚ 2946.9375 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.dm-0.out‚142315.9609375 disk.sda3.in‚1.96875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0% disk.nvme0c0n1.out‚142315.9609375$ memory_percent‚88.15343379974365 cpu‚25.565826660807318 network.recv‚ 2.340671e+06 disk.sda1.in‚0 disk.sda.in‚1.96875㭞Ƃ(:“ ðµëÒ–øÂÉ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚252.389' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.47780000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&‘]êÇ‚(:ß ðµëÒݹßË& disk.nvme0n1.out‚142320.0830078125 disk.sda.in‚1.96875 disk.drbd2165.in‚0# disk.dm-0.out‚142320.0830078125 proc.memory.rssMB‚ 2946.9375 network.sent‚ 1.0353175e+07 disk.sda3.in‚1.96875 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚88.15197944641113( proc.memory.availableMB‚ 1941.1796875$ disk./.usageGB‚593.9659652709961 disk.sda.out‚ 402.93359375 disk.sda1.out‚0 disk.sda3.out‚ 402.93359375 proc.cpu.threads‚17 network.recv‚ 2.345163e+06 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚142320.0830078125* proc.memory.percent‚17.986679077148438 disk.rbd0.out‚0 disk.rbd1.out‚ 25.1552734375 cpu‚25.566056869900073$ disk.nvme0c0n1.in‚26084.58203125) disk./.usagePercent‚39.86155429997628 disk.dm-0.in‚26084.58203125 disk.sda1.in‚0 disk.dm-1.in‚1.96875 disk.rbd0.in‚0 disk.dm-1.out‚ 402.93359375 disk.rbd1.in‚0" disk.nvme0n1.in‚26084.58203125Ô™C,ŸÈ‚(:” ÿµëÒ‘£ÉÇ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚334.587' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.91739999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁcšåÉ‚(:Ú ÿµëÒùè®É$ disk./.usageGB‚593.9704437255859 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 26085.546875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.out‚ 404.56640625 disk.rbd1.out‚ 25.2177734375" disk.nvme0c0n1.in‚ 26085.546875 disk.sda1.out‚0 disk.dm-1.in‚1.96875 disk.drbd2165.in‚0) disk./.usagePercent‚39.86185485282147' disk.nvme0n1.out‚142344.28466796875) disk.nvme0c0n1.out‚142344.28466796875* proc.memory.percent‚17.986679077148438# memory_percent‚88.1535291671753 disk.sda3.in‚1.96875 disk.sda2.in‚0$ disk.dm-0.out‚142344.28466796875 disk.nvme0n1.in‚ 26085.546875 disk.rbd0.in‚0) proc.memory.availableMB‚ 1940.92578125 network.recv‚ 2.34886e+06 disk.sda3.out‚ 404.56640625 disk.sda.in‚1.96875 proc.memory.rssMB‚ 2946.9375 network.sent‚ 1.0356097e+07 disk.rbd1.in‚0 disk.sda.out‚ 404.56640625 cpu‚25.56639103416031¤)a“Ê‚(:ˆ Ž¶ëÒÛÔƒÈ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚262.895' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.579 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03Ò¨5ÚË‚(:Ï Ž¶ëÒš¾ÓÉ disk.rbd1.in‚0( disk.nvme0c0n1.out‚142361.2255859375 disk.nvme0n1.in‚ 26086.4375 disk.rbd0.out‚0 disk.sda2.in‚0# memory_percent‚88.1535291671753# disk.dm-0.out‚142361.2255859375 disk.rbd0.in‚0 disk.drbd2165.in‚0 cpu‚25.56665578932405 disk.sda1.in‚0 disk.sda3.out‚ 408.4921875) proc.memory.availableMB‚ 1940.92578125 network.recv‚ 2.352314e+06) disk./.usagePercent‚39.86195136766188 disk.dm-1.in‚1.96875 disk.dm-1.out‚ 408.4921875 disk.nvme0c0n1.in‚ 26086.4375 disk.dm-0.in‚ 26086.4375& disk.nvme0n1.out‚142361.2255859375 disk.sda2.out‚0 disk.sda.in‚1.96875* proc.memory.percent‚17.986679077148438$ disk./.usageGB‚593.9718818664551 disk.drbd2165.out‚0 disk.sda1.out‚0 proc.memory.rssMB‚ 2946.9375 proc.cpu.threads‚17 network.sent‚ 1.0361126e+07 disk.rbd1.out‚ 25.2177734375 disk.sda.out‚ 408.4921875 disk.sda3.in‚1.96875þ°Ó”Ì‚(:‰ ¶ëÒÇ´¿É gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚288.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.7936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ >"âÍ‚(:× ¶ëÒà×ËË$ disk./.usageGB‚593.9645767211914' disk.nvme0n1.out‚142369.23779296875 disk.sda3.out‚ 411.171875 disk.rbd1.in‚0 disk.nvme0n1.in‚ 26157.359375 disk.sda1.out‚0 disk.sda3.in‚1.96875 disk.sda2.out‚0 disk.sda.out‚ 411.171875 disk.drbd2165.in‚0$ disk.dm-0.out‚142369.23779296875 disk.dm-1.in‚1.96875 disk.rbd1.out‚ 25.3427734375 disk.sda2.in‚0) disk.nvme0c0n1.out‚142369.23779296875 disk.dm-0.in‚ 26157.359375 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda.in‚1.96875 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚88.15500736236572) proc.memory.availableMB‚ 1940.68359375* proc.memory.percent‚17.986679077148438 cpu‚25.566883809475378 network.sent‚ 1.0368248e+07 network.recv‚ 2.355896e+06) disk./.usagePercent‚39.86146111323381 disk.dm-1.out‚ 411.171875" disk.nvme0c0n1.in‚ 26157.359375 proc.memory.rssMB‚ 2946.9375ÇÁŸ”΂(:‰ ¬¶ëÒö£ÍÇ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚333.378' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'ìØkÜÏ‚(:Ñ ¬¶ëÒðÆÖÉ) disk./.usagePercent‚39.86147365760299 disk.sda2.in‚0 proc.memory.rssMB‚ 2946.9375* proc.memory.percent‚17.986679077148438" disk.nvme0n1.in‚26158.32421875# memory_percent‚88.1535530090332' proc.memory.availableMB‚ 1940.921875! disk.dm-0.out‚142388.06640625 network.recv‚ 2.359171e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚1.96875 disk.dm-1.in‚1.96875 disk.sda.in‚1.96875& disk.nvme0c0n1.out‚142388.06640625$ disk./.usageGB‚593.9647636413574 disk.sda3.out‚ 413.515625 disk.dm-1.out‚ 413.515625 disk.drbd2165.in‚0 disk.dm-0.in‚26158.32421875 disk.rbd1.out‚ 25.3427734375 disk.sda.out‚ 413.515625 cpu‚25.567212416465356 network.sent‚ 1.0371324e+07 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚142388.06640625$ disk.nvme0c0n1.in‚26158.32421875 proc.cpu.threads‚17y€^2Ð- _wandbruntime‚9406  _runtime‚9406šš~OÉÑà % _timestamp‚1.7843392456376932e+09 _runtime‚9406.223984971 _step‚441 ' lr-AdamW/pg1‚0.00040908863362690815 ' lr-AdamW/pg2‚0.00040908863362690815 trainer/global_step‚5399Ý!ÈæÎÒÈ % _timestamp‚1.7843392456376932e+09 _runtime‚9406.223984971 _step‚441 ' lr-AdamW/pg1‚0.00040908863362690815 ' lr-AdamW/pg2‚0.00040908863362690815 trainer/global_step‚5399¹ ÷ƒ2Ó- _wandbruntime‚9406  _runtime‚9406vb'îþ'ÔøO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6544043556352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1550384906678272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1555453588382154 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5077369602560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚5399 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚9406.224116308 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46744307 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29187484 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚419244180131840 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1551055177326592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5078714992128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5078714992128 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1556132546929152 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 $ _timestamp‚1.784339245638417e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1550384932106240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46032849 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46754280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15334871 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5070026304204 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1551055203500032 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚442 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92777150 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92787129 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚425788260321792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚419243812040704 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5068681703882 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1556133918492160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29187204 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1556132546929152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1551055177326592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13852338 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46744307 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1556133918492160 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1551055203500032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13852333 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6544080189952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5077369602560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46754280 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46032843 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1555454958410444 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128‡.œ:ÊÕýO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6544043556352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1550384906678272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/requested_byte‰^û˜9%s.all.freed‚1555453588382154 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5077369602560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚5399 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚9406.224116308 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46744307 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29187484 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚419244180131840 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1551055177326592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5078714992128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5078714992128 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1556132546929152 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 $ _timestamp‚1.784339245638417e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1550384932106240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46032849 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46754280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15334871 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5070026304204 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1551055203500032 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚442 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92777150 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92787129 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚425788260321792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚419243812040704 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5068681703882 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1556133918492160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29187204 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1556132546929152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1551055177326592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13852338 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46744307 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1556133918492160 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1551055203500032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13852333 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6544080189952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5077369602560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46754280 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46032843 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1555454958410444 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440ºAeHÙ2Ö- _wandbruntime‚9407  _runtime‚9407ã's '׃N [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1550662788456448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29192462 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6545228561920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5078294638080 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29192727 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1556412844264960 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1551333178056704 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92803827 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚425866931727872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1551333204230144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1556411472694784 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5078294638080 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1556412844264960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1551333204230144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46041049 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚419321298448384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13854876 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15337846 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚425866527010304 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46041049 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1555733763654978 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _step‚443 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92793834 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6545265188352 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1550662813884416 trainer/global_step‚5399 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46752785 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843392470332713e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46041055 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46752785 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5079640034816 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚419321298448384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13854876 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15337846 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚425866527010304 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46041049 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1555733763654978 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _step‚443 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92793834 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6545265188352 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1550662813884416 trainer/global_step‚5399 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46752785 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843392470332713e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46041055 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46752785 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5079640034816 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29192462 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6545228561920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5078294638080 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29192727 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1556412844264960 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1551333178056704 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92803827 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚425866931727872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1551333204230144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1556411472694784 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5078294638080 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1556412844264960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0»¤ÉÅb2Ù- _wandbruntime‚9407  _runtime‚94079ñ&Úþ -loss/FRt2/detection_loss‚1.62620394358429 /loss/FBt2/detection_loss‚0.7724728316922361 epoch‚13 _step‚444 1loss/SRqqt2/assignment_loss‚0.558136106464242 1loss/FBt1/assignment_loss‚0.01236183593091688 /loss/FRt1/detection_loss‚0.5291973291889907 /loss/FBt1/detection_loss‚0.3629771172839763 %loss/total_loss‚8.169527119249787 % _timestamp‚1.7843392471160867e+09 0loss/FRt1/assignment_loss‚1.5763059078048471 /loss/FRt2/assignment_loss‚0.892143854555094 2loss/FBt2/assignment_loss‚0.026633768439393506 1loss/SRqqt1/detection_loss‚0.6936506860950901 trainer/global_step‚5399 1loss/SRqqt2/detection_loss‚0.7352669410560244 _runtime‚ 9407.70089069 2loss/SRqqt1/assignment_loss‚0.3841767971546857Ì(K Ûƒ 2loss/SRqqt1/assignment_loss‚0.3841767971546857 -loss/FRt2/detection_loss‚1.62620394358429 /loss/FBt2/detection_loss‚0.7724728316922361 epoch‚13 _step‚444 1loss/SRqqt2/assignment_loss‚0.558136106464242 1loss/FBt1/assignment_loss‚0.01236183593091688 /loss/FRt1/detection_loss‚0.5291973291889907 /loss/FBt1/detection_loss‚0.3629771172839763 %loss/total_loss‚8.169527119249787 % _timestamp‚1.7843392471160867e+09 0loss/FRt1/assignment_loss‚1.5763059078048471 /loss/FRt2/assignment_loss‚0.892143854555094 2loss/FBt2/assignment_loss‚0.026633768439393506 1loss/SRqqt1/detection_loss‚0.6936506860950901 trainer/global_step‚5399 1loss/SRqqt2/detection_loss‚0.7352669410560244 _runtime‚ 9407.70089069¼0‹×žÜ‚(:“ »¶ëÒ„ªöÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚269.905' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.98099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å2XßÝ‚(:Ô »¶ëÒŒ‡ÁÉ disk.sda3.in‚1.96875 disk.rbd0.out‚0( proc.memory.availableMB‚ 1940.8984375$ disk./.usageGB‚593.9646301269531 disk.sda1.out‚0 disk.dm-1.in‚1.96875 disk.rbd1.out‚ 25.4052734375 disk.sda.in‚1.96875 disk.drbd2165.in‚0$ memory_percent‚88.15369606018066* disk./.usagePercent‚39.861464697339294 disk.sda1.in‚0' disk.nvme0n1.out‚142399.61376953125 network.sent‚ 1.0375263e+07! disk.nvme0c0n1.in‚ 26159.15625 disk.dm-0.in‚ 26159.15625* proc.memory.percent‚17.986679077148438 cpu‚25.567424238762445 network.recv‚ 2.361801e+06 disk.sda2.out‚0 disk.drbd2165.out‚0$ disk.dm-0.out‚142399.61376953125 disk.sda2.in‚0) disk.nvme0c0n1.out‚142399.61376953125 disk.sda3.out‚ 414.734375 disk.dm-1.out‚ 414.734375 disk.rbd1.in‚0 disk.nvme0n1.in‚ 26159.15625 proc.memory.rssMB‚ 2946.9375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda.out‚ 414.734375- 4|Þ‚(:“ ʶëÒˆ¶¹É gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.meЫ¸"moryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚272.619' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.52380000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}D‡Påß‚(:Ú ʶëÒ圦Ë$ disk./.usageGB‚593.9638366699219 disk.sda3.in‚1.96875 disk.drbd2165.out‚0) disk./.usagePercent‚39.86141144777217 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 26159.15625 cpu‚25.567597907231946 disk.nvme0n1.in‚ 26159.15625 disk.sda1.in‚0 disk.rbd1.out‚ 25.4052734375 network.sent‚ 1.0440099e+07$ disk.dm-0.out‚142411.41455078125 disk.sda1.out‚0 disk.sda3.out‚ 416.66015625 disk.rbd0.out‚0 disk.sda.in‚1.96875) disk.nvme0c0n1.out‚142411.41455078125) proc.memory.availableMB‚ 1941.13671875 proc.memory.rssMB‚ 2946.9375 network.recv‚ 2.366429e+06 disk.dm-1.out‚ 416.66015625! disk.nvme0c0n1.in‚ 26159.15625$ memory_percent‚88.15224170684814* proc.memory.percent‚17.986679077148438 proc.cpu.threads‚17' disk.nvme0n1.out‚142411.41455078125 disk.sda2.out‚0 disk.dm-1.in‚1.96875 disk.rbd1.in‚0 disk.sda.out‚ 416.66015625 disk.drbd2165.in‚0¯ÂZ|Ÿà‚(:” Ù¶ëÒŠ„òÇ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚292.983' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.596599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê:ÈŒëá‚(:à Ù¶ëÒïäÞÉ" disk.nvme0n1.in‚26312.19140625 disk.rbd0.out‚0 disk.sda3.in‚1.96875 disk.sda3.out‚ 417.4296875 disk.rbd0.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚26312.19140625$ disk./.usageGB‚593.9648017883301 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 417.4296875 disk.sda.out‚ 417.4296875$ memory_percent‚88.15438747406006 network.recv‚ 2.369971e+06) disk./.usagePercent‚39.86147621767833 disk.dm-1.in‚1.96875) disk.nvme0c0n1.out‚142419.19677734375 network.sent‚ 1.0443063e+07 disk.drbd2165.in‚0* proc.memory.percent‚17.986679077148438 proc.cpu.threads‚17' disk.nvme0n1.out‚142419.19677734375 disk.sda1.in‚0 disk.rbd1.out‚ 25.4677734375 disk.sda.in‚1.96875 disk.rbd1.in‚0) proc.memory.availableMB‚ 1940.78515625 proc.memory.rssMB‚ 2946.9375 cpu‚25.567921842379388 disk.sda2.in‚0 disk.dm-0.in‚26312.19140625$ disk.dm-0.out‚142419.19677734375{¥¼1Ÿâ‚(:” è¶ëÒùпÇ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mp–ßã‚(:Ô è¶ëÒß…ÚÉ disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚1.96875 proc.cpu.threads‚17 disk.sda.in‚1.96875 disk.drbd2165.out‚0' disk.nvme0n1.out‚142426.90283203125 disk.sda3.in‚1.96875 disk.rbd1.out‚ 25.4677734375) proc.memory.availableMB‚ 1941.26171875 network.sent‚ 1.0448044e+07 disk.sda3.out‚418.125 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚26312.19140625" disk.nvme0n1.in‚26312.19140625 network.recv‚ 2.373067e+06 disk.sda2.in‚0$ memory_percent‚88.15147876739502 cpu‚25.568130810682472) disk./.usagePercent‚39.86147800973107 disk.dm-0.in‚26312.19140625$ disk.dm-0.out‚142426.90283203125 disk.dm-1.out‚418.125 disk.sda.out‚418.125* proc.memory.percent‚17.986679077148438$ disk./.usageGB‚593.9648284912109) disk.nvme0c0n1.out‚142426.90283203125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 proc.memory.rssMB‚ 2946.9375 disk.sda1.in‚0>ÈžŸä‚(:” ÷¶ëÒÏÿÛÉ gpu.0.gpu‚97.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚314.078' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚62.815599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0焪fÔå‚(:É ÷¶ëÒÀÞíË* proc.memory.percent‚17.986679077148438$ disk./.usageGB‚593.9650192260742 disk.dm-0.out‚ 142434.6875 disk.dm-1.in‚1.96875 disk.sda.out‚ 418.43359375 proc.cpu.threads‚17 cpu‚25.568458355229843 network.recv‚ 2.376626e+06 disk.nvme0n1.out‚ 142434.6875 disk.sda1.in‚0 disk.sda3.in‚1.96875$ memory_percent‚88.15145492553711 network.sent‚ 1.0453075e+07 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 26313.0234375' proc.memory.availableMB‚ 1941.265625 disk.dm-0.in‚ 26313.0234375 disk.rbd1.in‚0) disk./.usagePercent‚39.86149081010779 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 25.5302734375 disk.sda.in‚1.96875 proc.memory.rssMB‚ 2946.9375" disk.nvme0c0n1.out‚ 142434.6875 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 418.43359375# disk.nvme0c0n1.in‚ 26313.0234375 disk.sda1.out‚0 disk.sda3.out‚ 418.43359375)²T2æ- _wandbruntime‚9486  _runtime‚9486È Q6Éçà ' lr-AdamW/pg1‚0.00040769466352483865 ' lr-AdamW/pg2‚0.00040769466352483865 trainer/global_step‚5449 % _timestamp‚1.7843393256048348e+09 _runtime‚9486.189843554 _step‚445]Ãð+ÎèÈ ' lr-AdamW/pg2‚0.00040769466352483865 trainer/global_step‚5449 % _timestamp‚1.7843393256048348e+09 _runtime‚9486.189843554 _step‚445 ' lr-AdamW/pg1‚0.00040769466352483865½Ì#?ï2é- _wandbruntime‚9486  _runtime‚9486þÇ ÿ'êùO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29452532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5123685533184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1564955213832192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1564955213832192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13979483 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46443149 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29452252 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15472769 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1570080270928384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚5449 % _timestamp‚1.7843393256058197e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46443149 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5125030922752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5125030922752 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1569395284432352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚423118132424704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6603352653312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚446 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1564955240005632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚423118500515840 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46443143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1569393914404062 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1570080270928384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚429721889802752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46443143 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1564279021015040 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1564278995587072 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1570078899365376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1570078899365376 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5123685533184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5116263417312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93613202 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47180032 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1570080270928384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚429721889802752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46443143 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1564279021015040 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1564278995587072 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1570078899365376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1570078899365376 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5123685533184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5116263417312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93613202 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47180032 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1564955213832192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1564955213832192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13979483 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46443149 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29452252 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15472769 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1570080270928384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚5449 % _timestamp‚1.7843393256058197e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46443149 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5125030922752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5125030922752 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1569395284432352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚423118132424704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6603352653312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚446 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1564955240005632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚423118500515840 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46443143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926¾€±ô"2ì- _wandbruntime‚9487  _runtime‚9487šº8ÞíƒN REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47178537 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47188524 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1565233214562304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15475752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚429800154459648 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6604535627264 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29457514 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13982026 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93639879 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚429800559177216 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46451349 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46451349 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46451355 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1570359196701184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1570359196701184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5124610568704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5124610568704 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚423195986923520 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008Ãì.$'îˆN YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5125955965440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29457514 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13982026 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93639879 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚429800559177216 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46451349 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46451349 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46451355 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1570359196701184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1570359196701184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5124610568704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5124610568704 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚423195986923520 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47178537 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47188524 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1565233214562304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15475752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚429800154459648 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6604535627264 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 EÌþÆ _step‚449 & lr-AdamW/pg1‚0.0004062900021387037 & lr-AdamW/pg2‚0.0004062900021387037 trainer/global_step‚5499 % _timestamp‚1.7843394058457537e+09 _runtime‚9566.430821503Á>¹2ÿ- _wandbruntime‚9566  _runtime‚9566j1d®%€÷O ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5170001463808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14106638 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1578855276511232 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5170001463808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94459233 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6662723732992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5171346853376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5162500530420 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚5499 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46853449 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1584025251801600 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1578855250337792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46853443 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47605784 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1584026623364608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚426992820899840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _ruŽu[EOntime‚ 9566.43490363 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29717528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15610895 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5171346853376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119Îáë(üO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29717802 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1584026623364608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚426992820899840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 9566.43490363 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29717528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15610895 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5171346853376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5170001463808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14106638 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1578855276511232 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5170001463808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94459233 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6662723732992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5171346853376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5162500530420 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚5499 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46853449 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1584025251801600 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1578855250337792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46853443 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47605784 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29722784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6663912476160 trainer/global_step‚5499 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5172271896064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5170926499328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15613608 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5170926499328 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1579133251067904 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46861655 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚427069939216384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.784339407241725e+09 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1578450966274048 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚ 9567.82711157 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47604289 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46861655 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1579133277241344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1584304177567232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14109181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94465938 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1579133277241344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94475931 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1579133251067904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14109176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6663949102592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚451 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1584305549137408 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚427070307307520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46861649 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1578450991702016 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640·L1 '„†N NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47614276 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47614276 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1584305549137408 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94465938 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29723050 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚433734256410112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5170926499328 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1579133251067904 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46861655 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚427069939216384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.784339407241725e+09 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1578450966274048 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚ 9567.82711157 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47604289 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46861655 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1579133277241344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1584304177567232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14109181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94465938 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1579133277241344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94475931 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1579133251067904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14109176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6663949102592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚451 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1584305549137408 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚427070307307520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46861649 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1578450991702016 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29722784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6663912476160 trainer/global_step‚5499 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5172271896064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5170926499328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15613608 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B2…- _wandbruntime‚9567  _runtime‚9567RË= †… 2loss/FBt2/assignment_loss‚0.016282546396043214 /loss/FRt2/detection_loss‚1.6928645343212483 0loss/FBt1/detection_loss‚0.38017527404097984 %loss/total_loss‚7.802750329030966 epoch‚13 trainer/global_step‚5499 _runtime‚9567.908085727 _step‚452 3loss/SRqqt1/assignment_loss‚0.44868043322113504 1loss/SRqqt1/detection_loss‚0.6583698879747216 1loss/SRqqt2/detection_loss‚0.6351137991090479 % _timestamp‚1.7843394073233058e+09 /loss/FRt1/detection_loss‚0.5174614886404283 0loss/FBt2/detection_loss‚0.48232206665888067 .loss/FRt1/assignment_loss‚1.63999807258178 0loss/FRt2/assignment_loss‚0.7818759733148853 2loss/SRqqt2/assignment_loss‚0.5268832722365372 2loss/FBt1/assignment_loss‚0.022722980535278863®s¯w‡Š 3loss/SRqqt1/assignment_loss‚0.44868043322113504 1loss/SRqqt1/detection_loss‚0.6583698879747216 1loss/SRqqt2/detection_loss‚0.6351137991090479 % _timestamp‚1.7843394073233058e+09 /loss/FRt1/detection_loss‚0.5174614886404283 0loss/FBt2/detection_loss‚0.48232206665888067 .loss/FRt1/assignment_loss‚1.63999807258178 0loss/FRt2/assignment_loss‚0.7818759733148853 2loss/SRqqt2/assignment_loss‚0.5268832722365372 2loss/FBt1/assignment_loss‚0.022722980535278863 2loss/FBt2/assignment_loss‚0.016282546396043214 /loss/FRt2/detection_loss‚1.6928645343212483 0loss/FBt1/detection_loss‚0.38017527404097984 %loss/total_loss‚7.802750329030966 epoch‚13 trainer/global_step‚5499 _runtime‚9567.908085727 _step‚452Äj'æÞˆ‚(:‰ Ñ·ëÒ‚¿£Ì gpu.0.gpu‚41.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚335.402' gpu.0.enforb±/ضcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.0804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.*‹æ‰‚(:Û Ñ·ëÒæÃþÌ disk.sda3.in‚1.96875 disk.sda2.out‚0$ memory_percent‚88.15023899078369* proc.memory.percent‚17.986679077148438 disk.dm-1.in‚1.96875) disk.nvme0c0n1.out‚142488.38720703125 disk.sda3.out‚ 422.828125 disk.rbd0.in‚0) proc.memory.availableMB‚ 1941.46484375 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk.dm-0.out‚142488.38720703125" disk.nvme0n1.in‚26465.66796875 disk.sda1.in‚0 disk.sda.out‚ 422.828125$ disk.nvme0c0n1.in‚26465.66796875 disk.dm-0.in‚26465.66796875 disk.dm-1.out‚ 422.828125 disk.sda1.out‚0 network.sent‚ 1.055391e+07$ disk./.usageGB‚593.9604148864746) disk./.usagePercent‚39.86118180901396 disk.rbd1.out‚ 25.7802734375 cpu‚25.569972461894324 proc.cpu.threads‚17' disk.nvme0n1.out‚142488.38720703125 disk.rbd0.out‚0 disk.sda2.in‚0 proc.memory.rssMB‚ 2946.9375 network.recv‚ 2.39784e+06 disk.sda.in‚1.96875©zåV”Š‚(:‰ à·ëÒêë¸Ç gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚280.902' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šy‹ß‹‚(:Ô à·ëÒµ¾ƒÉ& disk.nvme0n1.out‚142504.1884765625 disk.sda3.out‚ 423.578125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda3.in‚1.96875 disk.sda.in‚1.96875 disk.dm-0.in‚ 26466.6328125( proc.memory.availableMB‚ 1941.1171875 cpu‚25.570093368448237) disk./.usagePercent‚39.86126040332697 disk.rbd0.in‚0 disk.rbd1.out‚ 25.7802734375 disk.drbd2165.out‚0# memory_percent‚88.1523609161377* proc.memory.percent‚17.986679077148438 proc.memory.rssMB‚ 2946.9375! disk.nvme0n1.in‚ 26466.6328125 disk.rbd0.out‚0 network.sent‚ 1.060381e+07$ disk./.usageGB‚593.9615859985352 disk.sda1.out‚0 disk.dm-1.in‚1.96875 disk.dm-1.out‚ 423.578125# disk.nvme0c0n1.in‚ 26466.6328125( disk.nvme0c0n1.out‚142504.1884765625# disk.dm-0.out‚142504.1884765625 network.recv‚ 2.402111e+06 disk.sda.out‚ 423.578125 disk.drbd2165.in‚0ßùŸŒ‚(:” ï·ëÒåçÞÈ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚299.744' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.948800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cªù¦ç‚(:Ü ï·ëÒ­–õÊ network.sent‚ 1.061107e+07$ disk./.usageGB‚593.9645919799805 disk.rbd1.in‚0 disk.nvme0n1.in‚ 26466.765625 disk.sda2.in‚0$ memory_percent‚88.15243244171143 network.recv‚ 2.405821e+06) disk./.usagePercent‚39.86146213726395 disk.sda.in‚1.96875 disk.sda3.in‚1.96875" disk.nvme0c0n1.in‚ 26466.765625 disk.dm-0.in‚ 26466.765625$ disk.dm-0.out‚142512.49365234375 disk.sda1.out‚0 proc.cpu.threads‚17) disk.nvme0c0n1.out‚142512.49365234375' disk.nvme0n1.out‚142512.49365234375 disk.sda1.in‚0 proc.memory.rssMB‚ 2946.9375 cpu‚25.570285366613557) proc.memory.availableMB‚ 1941.10546875 disk.dm-1.out‚ 425.53515625 disk.drbd2165.out‚0 disk.sda3.out‚ 425.53515625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚1.96875 disk.rbd1.out‚ 25.8427734375 disk.sda.out‚ 425.53515625 disk.drbd2165.in‚0* proc.memory.percent‚17.986679077148438 disk.sda2.out‚0´™3ë’Ž‚(:‡ þ·ëÒ‹ÉÍÇ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚348.41' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.682 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%þä‚(:Ù þ·ëÒéþÉ* proc.memory.percent‚17.986679077148438$ disk.nvme0c0n1.in‚26467.60546875$ disk.dm-0.out‚142531.43115234375 disk.sda2.out‚0) disk.nvme0c0n1.out‚142531.43115234375 disk.dm-0.in‚26467.60546875" disk.nvme0n1.in‚26467.60546875' disk.nvme0n1.out‚142531.43115234375 proc.cpu.threads‚17 network.recv‚ 2.409079e+06$ disk./.usageGB‚597.0309867858887 disk.sda.in‚1.96875 disk.sda.out‚ 426.96875 disk.sda2.in‚0 disk.dm-1.out‚ 426.96875# memory_percent‚88.1523847579956 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.sda3.out‚ 426.96875 cpu‚25.570621985317207 network.sent‚ 1.0613992e+07 disk.rbd1.out‚ 25.8427734375 disk.sda1.in‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1941.11328125) disk./.usagePercent‚40.06725046556506 disk.sda1.out‚0 disk.sda3.in‚1.96875 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚1.96875 proc.memory.rssMB‚ 2946.9375ŸíD Ÿ‚(:” ¸ëÒÿáÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.302' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.860400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;ÿ¯åâ‘‚(:× ¸ëÒåøîÈ network.sent‚ 1.061912e+07) disk./.usagePercent‚40.06722819290958 disk.sda2.in‚0 disk.sda.out‚ 428.6640625 disk.drbd2165.in‚0 network.recv‚ 2.412733e+06& disk.nvme0n1.out‚145675.4697265625 disk.rbd0.in‚0 cpu‚25.570829180728296 disk.dm-1.in‚1.96875# disk.nvme0c0n1.in‚ 26531.3671875( disk.nvme0c0n1.out‚145675.4697265625# disk.dm-0.out‚145675.4697265625 proc.cpu.threads‚17$ memory_percent‚88.15240859985352$ disk./.usageGB‚597.0306549072266 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 428.6640625 disk.sda.in‚1.96875 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 26531.3671875 disk.sda1.out‚0 disk.dm-0.in‚ 26531.3671875 disk.sda1.in‚0 proc.memory.rssMB‚ 2946.9375 disk.rbd1.in‚0 disk.rbd1.out‚ 25.9052734375 disk.sda3.in‚1.96875 disk.sda3.out‚ 428.6640625* proc.memory.percent‚17.986679077148438' proc.memory.availableMB‚ 1941.109375Y:N…”’‚(:‰ œ¸ëÒô¿ÆÉ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚320.961' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.1922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 C æ“‚(:Û œ¸ëÒü¥ÌË network.sent‚ 1.0626147e+07) disk./.usagePercent‚40.06716291098835 disk.rbd1.in‚0 disk.sda.in‚1.96875 disk.sda1.in‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1941.12109375 disk.dm-1.out‚ 431.3515625 disk.rbd1.out‚ 25.9052734375$ disk.dm-0.out‚145693.58544921875 disk.sda2.out‚0* proc.memory.percent‚17.986679077148438 cpu‚25.5711695695058) disk.nvme0c0n1.out‚145693.58544921875 disk.sda2.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 26583.2265625 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 26583.2265625 disk.dm-1.in‚1.96875 disk.sda.out‚ 431.3515625! disk.nvme0n1.in‚ 26583.2265625' disk.nvme0n1.out‚145693.58544921875 disk.sda3.in‚1.96875 disk.sda3.out‚ 431.3515625$ disk./.usageGB‚597.0296821594238 disk.drbd2165.in‚0 proc.cpu.threads‚17 network.recv‚ 2.416252e+06$ memory_percent‚88.15233707427979 proc.memory.rssMB‚ 2946.9375L}ug2”- _wandbruntime‚9646  _runtime‚9646ò"H¾Ç•Á & lr-AdamW/pg1‚0.0004048747396872794 & lr-AdamW/pg2‚0.0004048747396872794 trainer/global_step‚5549 % _timestamp‚1.7843394858247123e+09 _runtime‚9646.409777349 _step‚453iðWÌ–Æ & lr-AdamW/pg2‚0.0004048747396872794 trainer/global_step‚5549 % _timestamp‚1.7843394858247123e+09 _runtime‚9646.409777349 _step‚453 & lr-AdamW/pg1‚0.0004048747396872794Å8 ì-2—- _wandbruntime‚9646  _runtime‚9646ÁœÎÿ'˜ùO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1597274566447878 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5217662784000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5208737643528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47263743 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5216317394432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14233788 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _step‚454 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95285306 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48031536 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1597972975800832 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1597971604237824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47263749 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437589355040256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1597971604237824 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1592067198832640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14233783 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _runtime‚9646.411082052 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6722581847552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437589755570688 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _runtime‚9646.411082052 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6722581847552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437589755570688 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5217662784000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5208737643528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47263743 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5216317394432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14233788 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _step‚454 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95285306 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48031536 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1597972975800832 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1597971604237824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47263749 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437589355040256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1597971604237824 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1592067198832640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14233783 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5218587826688 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1593033313746944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5217242429952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5217242429952 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ]KDevice´/ì“ÆStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6723782985216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843394874127965e+09 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1598251901573632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1597554741720702 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47271955 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5218587826688 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29987096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48030041 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95311983 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48040028 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47271949 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚437668042585600 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47271949 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1598251901573632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430944627691520 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5209661109886 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995M´ 'œ‡N XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1593033287573504 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1592345055182848 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95301990 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48040028 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15750770 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95311983 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48040028 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47271949 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚437668042585600 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47271949 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1598251901573632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430944627691520 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5209661109886 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 _step‚455 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1598250530003456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5218587826688 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1593033313746944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5217242429952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5217242429952 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6723782985216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843394874127965e+09 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1598251901573632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1597554741720702 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47271955 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5218587826688 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29987096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48030041 F™Åã©‚(:Ø ç¸ëÒ’îÔÊ disk.sda2.out‚0 disk.sda.in‚1.96875 disk.drbd2165.in‚0$ disk.dm-0.out‚156565.20654296875 disk.rbd0.in‚0 network.recv‚ 2.434956e+06 disk.sda.out‚ 434.03125' disk.nvme0n1.out‚156565.20654296875 disk.sda3.out‚ 434.03125$ memory_percent‚88.15274238586426 disk.drbd2165.out‚0 network.sent‚ 1.0709288e+07 disk.dm-1.out‚ 434.03125 disk.dm-1.in‚1.96875 disk.rbd1.in‚0 disk.rbd1.out‚ 26.1552734375( proc.memory.availableMB‚ 1941.0546875( disk./.usagePercent‚40.7080155631281 disk.sda2.in‚0) disk.nvme0c0n1.out‚156565.20654296875* proc.memory.percent‚17.986679077148438$ disk.nvme0c0n1.in‚26665.63671875" disk.nvme0n1.in‚26665.63671875 disk.rbd0.out‚0 cpu‚25.572166214330938$ disk./.usageGB‚606.5788497924805 disk.sda3.in‚1.96875 disk.dm-0.in‚26665.63671875 disk.sda1.in‚0 disk.sda1.out‚0 proc.memory.rssMB‚ 2946.9375 proc.cpu.threads‚17Ë0œ—2ª- _wandbruntime‚9726  _runtime‚9726~«ÃÇ«Á & lr-AdamW/pg2‚0.0004034489670702283 trainer/global_step‚5599 % _timestamp‚1.7843395660288117e+09 _runtime‚9726.613839378 _step‚457 & lr-AdamW/pg1‚0.0004034489670702283ø)ØÌ¬Æ & lr-AdamW/pg1‚0.0004034489670702283 & lr-AdamW/pg2‚0.0004034489670702283 trainer/global_step‚5599 % _timestamp‚1.7843395660288117e+09 _runtime‚9726.613839378 _step‚457É0ñö2­- _wandbruntime‚9726  _runtime‚9726ä7’ø®ùO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1606655323348992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1605961262313472 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96121358 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _runtime‚9726.615122521 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47674049 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14360938 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434741093576704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47674043 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96131337 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30246420 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1611917956674048 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1611917956674048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5254974756636 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚441524265684480 _step‚458 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5262633325056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5253630156314 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48447315 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48457288 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5263978714624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1611919328237056 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1606655349522432 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434741461667840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6782771577344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5262633325056 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1606655323348992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4614144819“j6¸ 2 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1606655323348992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1605961262313472 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96121358 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _runtime‚9726.615122521 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47674049 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14360938 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434741093576704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47674043 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96131337 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30246420 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1611917956674048 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1611917956674048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5254974756636 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚441524265684480 _step‚458 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5262633325056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5253630156314 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48447315 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48457288 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5263978714624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1611919328237056 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1606655349522432 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434741461667840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6782771577344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5262633325056 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1606655323348992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5264903757312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1606933324079104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96138042 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96148035 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6784002202112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1606933324079104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48455793 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15888485 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1606239144091648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47682255 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5255898222994 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434818579984384 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5263558360576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1611493697714208 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96148035 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14363476 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47682255 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48455793 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1606933350252544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14363481 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15888257 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843395674240258e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1612198254009856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47682249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1606239169519616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30251733 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚441602549754368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47682249 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96138042 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚441602950277632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434818948075520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚459 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1606933350252544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6783969769984 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1612196882439680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5263558360576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1611495067742610 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5254553622560 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5264903757312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1606933324079104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96138042 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96148035 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6784002202112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1606933324079104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48455793 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15888485 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1606239144091648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47682255 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5255898222994 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434818579984384 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5263558360576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1611493697714208 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96148035 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14363476 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47682255 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48455793 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1606933350252544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14363481 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15888257 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843395674240258e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1612198254009856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47682249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1606239169519616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30251733 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚441602549754368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47682249 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96138042 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚441602950277632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434818948075520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 137157016’2w76 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚459 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1606933350252544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6783969769984 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1612196882439680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5263558360576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1611495067742610 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5254553622560 B&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.26632529497146606 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8283258080482483 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987881779670715 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9908525347709656 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.743127703666687 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.710454523563385 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.11265802383422852 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2771097421646118 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11970208585262299 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.09498032927513123 particle/accuracy_3_of_4‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 (loss/val_total_loss‚8.06344802972161 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9955822229385376 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.3370204269886017 particle/accuracy_2_of_3‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_4_of_5‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8769413828849792 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11496411263942719 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9991950988769531 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9904476404190063 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6388525366783142 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_2_of_4‚0.8064628839492798 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5905396342277527 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29493677616119385 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5692839622497559 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7529221773147583 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8719862103462219 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18098528683185577 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 5 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.56742262840271 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2815828025341034 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2683263421058655 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6180005073547363 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7238473296165466 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7856869697570801 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2530844509601593 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6966013312339783 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6377732157707214 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7181844115257263 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 epoch‚13 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6001323461532593 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.648982048034668 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6343720555305481 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969257116317749 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12916667759418488 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.271864652633667 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5591904520988464 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7074592113494873 )jet/accuracy_1_of_2‚0.794644296169281 +particle/sensitivity‚0.6432561278343201 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6707154512405396 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6551945805549622 *jet/accuracy_1_of_3‚0.9350509643554688 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2682896554470062 _runtime‚9755.174791536 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4156508147716522 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9748226404190063 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6095744967460632 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6197916269302368 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2952626645565033 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7930083274841309 7validation_average_jet_accuracy‚0.37753114104270935 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30354398488998413 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8275888562202454 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11496411263942719 trainer/global_step‚5599 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5935114622116089 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.12445332109928131 % _timestamp‚1.7843395945888524e+09 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.294299840927124 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5723651051521301 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35431891679763794 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.25567880272865295 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_1_of_1‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11880868673324585 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7207104563713074 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 (particle/accuracy‚0.8517679572105408 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0649426057934761 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.34488797187805176 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6396923065185547 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.671419620513916 +jet/accuracy_2_of_2‚0.15716981887817383 jet/accuracy_5_of_5‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6295290589332581 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6439858675003052 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6578792333602905 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7162379622459412 particle/accuracy_1_of_2‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21662533283233643 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04683300107717514 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 _step‚461 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9328185319900513 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30954214930534363 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.339277446269989 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4745216965675354 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1212669089436531 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.828873872756958 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6208019256591797 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05877923220396042 particle/accuracy_2_of_2‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4792872965335846 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48320451378822327 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 +jet/accuracy_1_of_1‚0.33494001626968384 +particle/specificity‚0.9087973833084106 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7856869697570801 particle/accuracy_4_of_5‚1 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1843479871749878 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28444424271583557 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28444424271583557 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 particle/accuracy_1_of_4‚1 'particle/f_score‚0.6508110761642456 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8947892189025879 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6683047413825989 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.468423992395401 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6331936717033386 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 /particle/accuracy_4_of_4‚0.9517605304718018 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5908455848693848 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2786114513874054 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5358333587646484 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9963633418083191 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9969407916069031 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6681941151618958 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6424490213394165 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.40398338437080383 +jet/accuracy_3_of_3‚0.07247097790241241 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28915801644325256 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28360697627067566 particle/accuracy_3_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9964457750320435 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2683952748775482 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27896153926849365 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6864535212516785 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.24240423738956451 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6349843144416809 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3229166567325592ÅÝ^îf-¼òz @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5591904520988464 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7074592113494873 )jet/accuracy_1_of_2‚0.794644296169281 +particle/sensitivity‚0.6432561278343201 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6707154512405396 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6551945805549622 *jet/accuracy_1_of_3‚0.9350509643554688 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2682896554470062 _runtime‚9755.174791536 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4156508147716522 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9748226404190063 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6095744967460632 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6197916269302368 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2952626645565033 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7930083274841309 7validation_average_jet_accuracy‚0.37753114104270935 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30354398488998413 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8275888562202454 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11496411263942719 trainer/global_step‚5599 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5935114622116089 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.12445332109928131 % _timestamp‚1.7843395945888524e+09 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.294299840927124 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5723651051521301 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35431891679763794 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.25567880272865295 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_1_of_1‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11880868673324585 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7207104563713074 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 (particle/accuracy‚0.8517679572105408 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0649426057934761 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.34488797187805176 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6396923065185547 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.671419620513916 +jet/accuracy_2_of_2‚0.15716981887817383 jet/accuracy_5_of_5‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6295290589332581 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6439858675003052 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6578792333602905 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7162379622459412 particle/accuracy_1_of_2‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21662533283233643 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04683300107717514 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 _step‚461 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9328185319900513 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30954214930534363 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.339277446269989 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4745216965675354 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1212669089436531 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.828873872756958 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6208019256591797 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05877923220396042 particle/accuracy_2_of_2‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4792872965335846 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48320451378822327 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 +jet/accuracy_1_of_1‚0.33494001626968384 +particle/specificity‚0.9087973833084106 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7856869697570801 particle/accuracy_4_of_5‚1 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1843479871749878 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28444424271583557 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28444424271583557 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 particle/accuracy_1_of_4‚1 'particle/f_score‚0.6508110761642456 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8947892189025879 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6683047413825989 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.468423992395401 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6331936717033386 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 /particle/accuracy_4_of_4‚0.9517605304718018 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5908455848693848 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2786114513874054 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5358333587646484 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9963633418083191 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9969407916069031 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6681941151618958 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6424490213394165 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.40398338437080383 +jet/accuracy_3_of_3‚0.07247097790241241 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28915801644325256 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28360697627067566 particle/accuracy_3_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9964457750320435 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2683952748775482 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27896153926849365 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6864535212516785 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.24240423738956451 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6349843144416809 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3229166567325592 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8249055743217468 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3598313629627228 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5908455848693848 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3407139480113983 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11981751769781113 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 jet/accuracy_2_of_5‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6009236574172974 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05789284408092499 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6917392015457153 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.608913779258728 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22582514584064484 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.26632529497146606 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8283258080482483 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987881779670715 7 Purity/0FRt*'ä½SRqqt2FBt/FBt_purity‚0.9908525347709656 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.743127703666687 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.710454523563385 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.11265802383422852 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2771097421646118 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11970208585262299 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.09498032927513123 particle/accuracy_3_of_4‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 (loss/val_total_loss‚8.06344802972161 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9955822229385376 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.3370204269886017 particle/accuracy_2_of_3‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_4_of_5‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8769413828849792 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11496411263942719 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9991950988769531 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9904476404190063 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6388525366783142 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_2_of_4‚0.8064628839492798 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5905396342277527 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29493677616119385 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5692839622497559 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7529221773147583 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8719862103462219 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18098528683185577 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 5 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.56742262840271 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2815828025341034 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2683263421058655 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6180005073547363 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7238473296165466 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7856869697570801 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2530844509601593 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6966013312339783 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6377732157707214 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7181844115257263 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 epoch‚13 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6001323461532593 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.648982048034668 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6343720555305481 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969257116317749 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12916667759418488 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.271864652633667ÍI2\Ã’½‚(:‡ ”¹ëÒËŽÊ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚287.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u…Ïᾂ(:Ö ”¹ëÒçÀ‰Ì network.sent‚ 1.0792878e+07 disk.rbd1.in‚0 disk.rbd1.out‚ 26.2177734375 network.recv‚ 2.445676e+06) disk./.usagePercent‚40.71737391854226 disk.sda3.out‚ 439.328125 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda3.in‚1.96875 disk.sda2.out‚0 disk.dm-1.out‚ 439.328125" memory_percent‚82.192063331604 cpu‚25.577853178766933& disk.nvme0n1.out‚157831.9951171875 disk.dm-1.in‚1.96875 disk.dm-0.in‚ 27606.515625 disk.nvme0n1.in‚ 27606.515625 disk.sda.in‚1.96875 disk.sda1.out‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚157831.9951171875# proc.memory.rssMB‚ 2950.06640625$ disk./.usageGB‚606.7182960510254 disk.sda1.in‚0 disk.sda.out‚ 439.328125) proc.memory.availableMB‚ 2917.65234375* proc.memory.percent‚18.005776405334473 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 27606.515625# disk.dm-0.out‚157831.9951171875óûÿÆž¿‚(:“ £¹ëÒ•¾ÀÉ gpu.0.gpu‚86.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚338.745' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.74900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú©s[ìÀ‚(:á £¹ëÒó’ºË disk.sda2.in‚0" disk.nvme0c0n1.in‚ 27607.484375# proc.memory.rssMB‚ 2950.06640625* proc.memory.percent‚18.005776405334473 cpu‚25.578170979691993$ disk./.usageGB‚606.7800102233887 disk.nvme0n1.in‚ 27607.484375' disk.nvme0n1.out‚157839.76123046875 disk.sda3.out‚ 439.74609375) proc.memory.availableMB‚ 2917.89453125 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚157839.76123046875 network.recv‚ 2.450454e+06 disk.sda1.in‚0 disk.sda.in‚1.96875 disk.drbd2165.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 439.74609375 disk.sda.out‚ 439.74609375 disk.dm-1.in‚1.96875) disk./.usagePercent‚40.72151560843119 disk.dm-0.in‚ 27607.484375$ disk.dm-0.out‚157839.76123046875 proc.cpu.threads‚17 network.sent‚ 1.0837193e+07 disk.sda3.in‚1.96875 disk.rbd0.out‚0 disk.rbd1.out‚ 26.2802734375$ memory_percent‚82.19058513641357kŸÁ‚(:” ²¹ëÒ¿ñ©É gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚266.683' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.336600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þn•î‚(:ã ²¹ëÒŠºÃÉ disk.sda3.out‚ 440.16015625 disk.rbd0.out‚0 disk.dm-1.in‚1.96875 disk.rbd1.in‚0$ disk.nvme0c0n1.in‚27623.16796875 proc.cpu.threads‚17 network.sent‚ 1.0842258e+07& disk.nvme0n1.out‚158886.6005859375# disk.dm-0.out‚158886.6005859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 26.2802734375$ memory_percent‚82.19208717346191# proc.memory.rssMB‚ 2950.06640625* proc.memory.percent‚18.005776405334473 disk.sda2.out‚0 disk.dm-1.out‚ 440.16015625 disk.drbd2165.in‚0 network.recv‚ 2.453673e+06$ disk./.usageGB‚606.7766571044922 disk.sda.in‚1.96875 disk.sda.out‚ 440.16015625 cpu‚25.57831667010057 disk.rbd0.in‚0 disk.dm-0.in‚27623.16796875" disk.nvme0n1.in‚27623.16796875 disk.sda2.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚158886.6005859375( proc.memory.availableMB‚ 2917.6484375* disk./.usagePercent‚40.721290577808595 disk.sda3.in‚1.96875SãK“Â(:ˆ Á¹ëÒººÙÉ gpu.0.gpu‚19.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡ÎïáÄ‚(:Ö Á¹ëÒ¢½ìË disk.dm-0.in‚ 27800.0625 cpu‚25.578616650616773 disk.sda2.out‚0 disk.sda.in‚1.96875$ disk.dm-0.out‚158891.10400390625# proc.memory.rssMB‚ 2950.06640625 disk.nvme0n1.in‚ 27800.0625 disk.rbd0.out‚0 disk.sda.out‚ 440.5390625$ memory_percent‚82.19213485717773 network.recv‚ 2.456417e+06 disk.rbd0.in‚0 disk.dm-1.in‚1.96875* proc.memory.percent‚18.005776405334473 proc.cpu.threads‚17 network.sent‚ 1.0848324e+07# disk./.usageGB‚606.776985168457 disk.sda3.out‚ 440.5390625 disk.sda2.in‚0' proc.memory.availableMB‚ 2917.640625 disk.dm-1.out‚ 440.5390625 disk.drbd2165.in‚0 disk.sda1.out‚0 disk.sda3.in‚1.96875 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 27800.0625* disk./.usagePercent‚40.721312594456535' disk.nvme0n1.out‚158891.10400390625 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 26.3427734375) disk.nvme0c0n1.out‚158891.10400390625ƒ0ŸÅ‚(:” йëÒÒÐÇ gpu.0.gpu‚89.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚295.399' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.079800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„ œðÆ‚(:å йëÒ»íÏÉ disk.drbd2165.in‚0( disk.nvme0c0n1.out‚158970.3427734375 disk.sda1.in‚0 disk.dm-1.out‚ 440.85546875 disk.sda.in‚1.96875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 440.85546875& disk.nvme0n1.out‚158970.3427734375 disk.rbd1.out‚ 26.3427734375 cpu‚25.578788172638813 proc.cpu.threads‚17 network.recv‚ 2.459678e+06$ disk./.usageGB‚606.7769203186035 disk.sda3.out‚ 440.85546875 disk.sda2.out‚0$ memory_percent‚82.19215869903564 disk.drbd2165.out‚0# disk.dm-0.out‚158970.3427734375) proc.memory.availableMB‚ 2917.63671875* disk./.usagePercent‚40.721308242328455 disk.sda1.out‚0* proc.memory.percent‚18.005776405334473 network.sent‚ 1.0851246e+07 disk.dm-0.in‚27801.02734375" disk.nvme0n1.in‚27801.02734375 disk.sda3.in‚1.96875 disk.sda2.in‚0 disk.rbd1.in‚0$ disk.nvme0c0n1.in‚27801.02734375 disk.dm-1.in‚1.96875# proc.memory.rssMB‚ 2950.06640625Šõ£2Ç- _wandbruntime‚9838  _runtime‚98389A£ ÉÈà ' lr-AdamW/pg2‚0.00040201277586226136 trainer/global_step‚5649 % _timestamp‚1.7843396779550154e+09 _runtime‚9838.540150539 _step‚462 ' lr-AdamW/pg1‚0.00040201277586226136²A$¬ÎÉÈ ' lr-AdamW/pg1‚0.00040201277586226136 ' lr-AdamW/pg2‚0.00040201277586226136 trainer/global_step‚5649 % _timestamp‚1.7843396779550154e+09 _runtime‚9838.540150539 _step‚462Î…~¹Ý2Ê- _wandbruntime‚9838  _runtime‚9838LÎNœÿ'ËùO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1624383367462912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1629699504722432 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1628988326584034 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14497006 % _timestamp‚1.7843396779562368e+09 _step‚463 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48216355 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6849920591872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48942027 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1629698133159424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1624383367462912 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚438663894298624 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚5649 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48216349 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5314791869952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5316137259520 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚445513847329792 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1623681260154880 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5316137259520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48216349 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1623681285582848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1624383341289472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48932054 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30552918 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1624383341289472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16055917 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚438663526207488 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1629698133159424 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48932054 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97158382 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6849953031168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16056156 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚445513446799360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 Y>G(ÌþO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48216355 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6849920591872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48942027 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1629698133159424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1624383367462912 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚438663894298624 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚5649 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48216349 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5314791869952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5316137259520 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚445513847329792 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1623681260154880 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5316137259520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48216349 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1623681285582848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1624383341289472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48932054 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30552918 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1624383341289472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16055917 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚438663526207488 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1629698133159424 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48932054 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97158382 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6849953031168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16056156 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚445513446799360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1624383367462912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1629699504722432 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1628988326584034 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14497006 % _timestamp‚1.7843396779562368e+09 _step‚463 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648Ï_ZtžÍ‚(:“ ß¹ëÒ¾‰ÐÈ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚279.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.85099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íà"Ô@΂(:Û ß¹ëÒûžòÉ disk.sda.out‚ 441.4765625% disk.Or -¦nvme0n1.out‚158974.154296875 proc.cpu.threads‚17( disk./.usagePercent‚40.7213586758127 disk.dm-1.in‚1.96875 disk.rbd1.out‚ 26.4052734375 disk.sda.in‚1.96875 disk.sda3.out‚ 441.4765625 disk.sda2.in‚0 cpu‚25.578945675137724 network.sent‚ 1.0856348e+07 disk.rbd1.in‚0 disk.dm-0.in‚27801.02734375 disk.sda1.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2950.06640625$ disk./.usageGB‚606.7776718139648 disk.drbd2165.out‚0" disk.dm-0.out‚158974.154296875 disk.rbd0.out‚0( proc.memory.availableMB‚ 2917.6484375 network.recv‚ 2.46316e+06 disk.sda1.out‚0* proc.memory.percent‚18.005776405334473 disk.sda2.out‚0$ memory_percent‚82.19208717346191 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚27801.02734375" disk.nvme0n1.in‚27801.02734375 disk.dm-1.out‚ 441.4765625' disk.nvme0c0n1.out‚158974.154296875 disk.sda3.in‚1.96875©´·2Ï- _wandbruntime‚9840  _runtime‚9840È»zd'ЂN _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1624661368193024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1623959141933056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5649 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1624661368193024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6851163213824 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5306619867110 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5315716905472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5307964467544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48224561 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6851130781696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1629978430495232 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16058859 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1629978430495232 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97165087 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48940532 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1624661342019584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48950519 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1623959167361024 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30558175 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5315716905472 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97175080 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30558408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97175080 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚438741012615168 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚445592143396864 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0¶ï 'чN >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48940532 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1624661342019584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48950519 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1623959167361024 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30558175 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5315716905472 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97175080 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30558408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97175080 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚438741012615168 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚445592143396864 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1624661368193024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1623959141933056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5649 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1624661368193024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6851163213824 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5306619867110 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5315716905472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5307964467544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48224561 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6851130781696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1629978430495232 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16058859 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1629978430495232 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97165087ÐùGQ2Ò- _wandbruntime‚9840  _runtime‚9840É, Ó„ 3loss/SRqqt1/assignment_loss‚0.43525013884046554 /loss/FRt2/detection_loss‚1.6970495200133011 /loss/FBt2/detection_loss‚0.5082138106010334 _step‚465 %loss/total_loss‚8.147892557123567 epoch‚14 0loss/FRt1/assignment_loss‚1.5641808986114467 1loss/FBt2/assignment_loss‚0.01646824984850021 1loss/SRqqt1/detection_loss‚0.7380998306342895 /loss/FBt1/detection_loss‚0.6153728713092517 2loss/FBt1/assignment_loss‚0.034673580944726314 % _timestamp‚1.7843396796455202e+09 0loss/FRt2/assignment_loss‚0.7996151698497763 2loss/SRqqt2/assignment_loss‚0.5305741377301266 /loss/FRt1/detection_loss‚0.5160137426872266 1loss/SRqqt2/detection_loss‚0.6923806060534242 trainer/global_step‚5649 _runtime‚9840.230302278x†¡$Ô‰ _runtime‚9840.230302278 3loss/SRqqt1/assignment_loss‚0.43525013884046554 /loss/FRt2/detection_loss‚1.6970495200133011 /loss/FBt2/detection_loss‚0.5082138106010334 _step‚465 %loss/total_loss‚8.147892557123567 epoch‚14 0loss/FRt1/assignment_loss‚1.5641808986114467 1loss/FBt2/assignment_loss‚0.01646824984850021 1loss/SRqqt1/detection_loss‚0.7380998306342895 /loss/FBt1/detection_loss‚0.6153728713092517 2loss/FBt1/assignment_loss‚0.034673580944726314 % _timestamp‚1.7843396796455202e+09 0loss/FRt2/assignment_loss‚0.7996151698497763 2loss/SRqqt2/assignment_loss‚0.5305741377301266 /loss/FRt1/detection_loss‚0.5160137426872266 1loss/SRqqt2/detection_loss‚0.6923806060534242 trainer/global_step‚5649Ñ7ˆßÕ‚(:’ î¹ëÒ©¥ÛË gpu.0.gpu‚86.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚331.35' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.27000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VòHäÖ‚(:Ù î¹ëÒ’¬¬Ì( proc.memory.availableMB‚ 2917.8515625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 443.484375 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk.dm-0.out‚159052.18115234375' disk.nvme0n1.out‚159052.18115234375 disk.dm-1.in‚1.96875$ memory_percent‚82.19084739685059 disk.rbd0.in‚0* proc.memory.percent‚18.005776405334473" disk.nvme0c0n1.in‚ 27801.859375 disk.sda1.out‚0 disk.dm-1.out‚ 443.484375 cpu‚25.579217405301158* disk./.usagePercent‚40.238895581039614# disk./.usageGB‚599.588623046875 disk.sda3.in‚1.96875 disk.sda3.out‚ 443.484375 disk.rbd1.out‚ 26.4052734375 disk.nvme0n1.in‚ 27801.859375 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2950.06640625 network.recv‚ 2.46751e+06 disk.sda.in‚1.96875) disk.nvme0c0n1.out‚159052.18115234375 network.sent‚ 1.0908364e+07 disk.dm-0.in‚ 27801.859375 disk.rbd0.out‚0²½=J”ׂ(:‰ ý¹ëÒÓÇäÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚257.222' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.4444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0— çØ‚(:Ü ý¹ëÒ¯µñÉ network.sent‚ 1.0960664e+07 disk.rbd0.out‚0 disk.drbd2165.in‚0$ disk.nvme0n1.out‚159122.51953125$ memory_percent‚82.19242095947266* proc.memory.percent‚18.005776405334473 disk.sda.out‚ 444.16796875# proc.memory.rssMB‚ 2950.06640625 disk.sda3.in‚1.96875 disk.drbd2165.out‚0 disk.dm-1.out‚ 444.16796875! disk.dm-0.out‚159122.51953125 network.recv‚ 2.472228e+06 disk.sda2.in‚0 disk.rbd1.out‚ 26.4677734375 disk.sda1.in‚0& proc.memory.availableMB‚ 2917.59375$ disk./.usageGB‚599.5883483886719 disk.rbd0.in‚0" disk.nvme0n1.in‚27848.88671875 disk.sda1.out‚0 cpu‚25.579388515944856 disk.sda3.out‚ 444.16796875 disk.dm-1.in‚1.96875 disk.rbd1.in‚0 disk.sda.in‚1.96875$ disk.nvme0c0n1.in‚27848.88671875& disk.nvme0c0n1.out‚159122.51953125 disk.dm-0.in‚27848.88671875 proc.cpu.threads‚17* disk./.usagePercent‚40.238877148497146 disk.sda2.out‚0\(¼ÁžÙ‚(:“ ŒºëÒ‡öÊÉ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚291.631' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.32619999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èÍ+ëÚ‚(:à ŒºëÒŸíË( proc.memory.availableMB‚ 2917.6015625 network.recv‚ 2.475371e+06" disk.nvme0n1.in‚27939.51953125 disk.sda3.out‚ 444.953125$ disk.dm-0.out‚159191.53271484375 proc.cpu.threads‚17$ disk./.usageGB‚599.5883026123047 disk.dm-1.in‚1.96875 disk.sda.in‚1.96875$ disk.nvme0c0n1.in‚27939.51953125 disk.rbd1.in‚0) disk.nvme0c0n1.out‚159191.53271484375# proc.memory.rssMB‚ 2950.06640625) disk./.usagePercent‚40.23887407640674 disk.sda1.in‚0 disk.sda3.in‚1.96875 disk.rbd1.out‚ 26.4677734375 disk.drbd2165.in‚0* proc.memory.percent‚18.005776405334473 network.sent‚ 1.0967786e+07' disk.nvme0n1.out‚159191.53271484375 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 444.953125 disk.sda.out‚ 444.953125 disk.dm-0.in‚27939.51953125 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚82.19237327575684 cpu‚25.579585716322047']N”Û‚(:‰ ›ºëÒôĺÈ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚331.202' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.2404 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€Y‹XÙÜ‚(:Î ›ºëÒ¶ŸþÈ network.sent‚ 1.097075e+07 disk.rbd0.out‚0 disk.dm-1.in‚1.96875 disk.rbd1.out‚ 26.5927734375 disk.drbd2165.out‚0' disk.nvme0n1.out‚159199.73779296875 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda3.out‚445.25 disk.sda.out‚445.25) disk./.usagePercent‚40.23988556217452# disk.nvme0c0n1.in‚ 27940.3515625$ disk./.usageGB‚599.6033744812012) disk.nvme0c0n1.out‚159199.73779296875 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚1.96875& proc.memory.availableMB‚ 2917.59375* proc.memory.percent‚18.005776405334473! disk.nvme0n1.in‚ 27940.3515625 cpu‚25.579883876635513 disk.rbd0.in‚0 disk.dm-1.out‚445.25 disk.sda.in‚1.96875 disk.drbd2165.in‚0 disk.dm-0.in‚ 27940.3515625 network.recv‚ 2.478765e+06 disk.sda2.in‚0 disk.rbd1.in‚0$ disk.dm-0.out‚159199.73779296875$ memory_percent‚82.19242095947266# proc.memory.rssMB‚ 2950.06640625€]E”Ý‚(:‰ ªºëÒì¤ÊÇ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤}vèÞ‚(:Ý ªºëÒ‰úÜÉ network.recv‚ 2.481808e+06! disk.dm-0.out‚159288.07421875 disk.dm-1.in‚1.96875 proc.cpu.threads‚17& disk.nvme0c0n1.out‚159288.07421875" disk.nvme0n1.in‚27941.31640625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0* proc.memory.percent‚18.006086349487305) proc.memory.availableMB‚ 2917.30078125$ disk./.usageGB‚599.5269737243652 disk.sda.out‚ 445.55078125 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚27941.31640625 disk.rbd0.in‚0 disk.rbd1.out‚ 26.5927734375 network.sent‚ 1.0977763e+07* disk./.usagePercent‚40.234758243278755 disk.sda.in‚1.96875 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.sda3.in‚1.96875 disk.dm-1.out‚ 445.55078125$ memory_percent‚82.19420909881592" proc.memory.rssMB‚ 2950.1171875$ disk.nvme0n1.out‚159288.07421875 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.58008629249062 disk.dm-0.in‚27941.31640625 disk.sda3.out‚ 445.55078125lVµZ2ß- _wandbruntime‚9918  _runtime‚9918¸€S\ÆàÀ trainer/global_step‚5699 % _timestamp‚1.7843397580928636e+09 _runtime‚ 9918.67787525 _step‚466 & lr-AdamW/pg1‚0.0004005662583072561 & lr-AdamW/pg2‚0.0004005662583072561u1ËáÅ & lr-AdamW/pg1‚0.0004005662583072561 & lr-AdamW/pg2‚0.0004005662583072561 trainer/global_step‚5699 % _timestamp‚1.7843397580928636e+09 _runtime‚ 9918.67787525 _step‚466Ò(ÀHV2â- _wandbruntime‚9918  _runtime‚9918TÔ†Ë ãùO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48626649 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1643644485595648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1643644485595648 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1638283377795072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30817990 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4754² 45124352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6909421096960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5699 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚9918.681848571 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14624156 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16193839 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449447231054848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚442537846591488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1642927282577652 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49357806 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1637575374491648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5361107800576 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚442538214682624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49357806 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1643645857158656 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1643645857158656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30818262 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97984455 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚467 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14624151 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97994434 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16194106 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1642928652605942 % _timestamp‚1.7843397580938892e+09 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1638283377795072 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5353278114294 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5362453190144 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5362453190144 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1643644485595648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1643644485595648 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1638283377795072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30817990 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6909421096960 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5699 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚9918.681848571 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14624156 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16193839 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449447231054848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚442537846591488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1642927282577652 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49357806 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1637575374491648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5361107800576 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚442538214682624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49357806 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1643645857158656 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1643645857158656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30818262 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97984455 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚467 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14624151 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97994434 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16194106 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1642928652605942 % _timestamp‚1.7843397580938892e+09 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1638283377795072 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5353278114294 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1637853230841856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5352856980218 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 _step‚468 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98011132 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1638561404698624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚9920.071911983 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1638561404698624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚449526308946944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6910607856640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30823271 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚442615701090304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1637853256269824 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5354201580652 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48634861 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16196577 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14626694 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1643206087822074 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49366284 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1638561378525184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5362032836096 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1643923411361280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449525904229376 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48634861 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5363378232832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98001139 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98011132 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128`$6‚jçˆN K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98011132 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16196833 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6910571230208 trainer/global_step‚5699 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48634855 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1637853230841856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5352856980218 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 _step‚468 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98011132 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1638561404698624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚9920.071911983 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1638561404698624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚449526308946944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6910607856640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30823271 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚442615701090304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1637853256269824 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ;5DeviceStatsMonitor.on_train_bat±Ê±ƒ¤ch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5354201580652 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48634861 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16196577 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14626694 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1643206087822074 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49366284 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1638561378525184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5362032836096 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1643923411361280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449525904229376 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48634861 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5363378232832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98001139 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1657590838031872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1656868978627850 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1657592209594880 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1651469437972480 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1652183414300672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1657590838031872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49793531 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49793531 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5408769120768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5407423731200 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1652183440474112 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚446412535066624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚446412166975488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5399515227402 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31083262 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49036955 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5398170627080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16331701 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 trainer/global_step‚5749 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49036949 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 % _timestamp‚1.7843398382798376e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16331956 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1651469463400448 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49036949 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚453380742115328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6968609676288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1657592209594880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1656867608599560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98830486 _step‚471 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1652183414300672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440—­‹d úýO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49793531 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5408769120768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5407423731200 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1652183440474112 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚446412535066624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚446412166975488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5399515227402 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31083262 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49036955 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5398170627080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16331701 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 trainer/global_step‚5749 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49036949 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 % _timestamp‚1.7843398382798376e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16331956 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1651469463400448 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49036949 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚453380742115328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6968609676288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1657592209594880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1656867608599560 F@DeviceStatsMonitor.on_trê×U øain_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98830486 _step‚471 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1652183414300672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1657590838031872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1656868978627850 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1657592209594880 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1651469437972480 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1652183414300672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1657590838031872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49793531 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820×üªìu4û/ _wandbruntime‚10000  _runtime‚10000°çÎô'üN [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1652461415030784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49045155 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31088282 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1651747319750656 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98847184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1657869763797504 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49792036 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98847184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 trainer/global_step‚5749 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16334438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1652461441204224 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49045155 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98837191 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1652461415030784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6969797002240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5408348766720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5400438693760 " _timestamp‚1.7843398396748e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446489653383168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31088532 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5409694163456 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5408348766720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚453459415856128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚10000.260080852 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1657871135367680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49045161 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49802023 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008¸~Šâ 'ý†N F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚453459818476544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 _step‚472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1657871135367680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49045161 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49802023 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1652461415030784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49045155 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31088282 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1651747319750656 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98847184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1657869763797504 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49792036 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98847184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 trainer/global_step‚5749 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16334438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1652461441204224 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49045155 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98837191 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1652461415030784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6969797002240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5408348766720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5400438693760 " _timestamp‚1.7843398396748e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446489653383168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31088532 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5409694163456 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5408348766720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚453459415856128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚10000.260080852 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16467561 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7027753692672 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1665363526881280 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50209310 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5455085051392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31345745 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1670807934621468 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1671538562031104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5453739661824 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457314645776896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1666083450806272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1671537190468096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚450286487359488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14878456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1671537190468096 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚450286855450624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1665363552309248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1666083476979712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99656559 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1666083476979712 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49447249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚5799¬ÿ¸ú(þO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5453739661824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1671538562031104 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5445752340510 _step‚475 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1666083450806272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50209310 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50219283 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49447255 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50219283 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5444407740188 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457314241052160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49447249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚5799 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49447249 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7027790326272 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31346017 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16467294 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 _runtime‚10078.964937855 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16467561 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7027753692672 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1665363526881280 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50209310 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5455085051392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31345745 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1670807934621468 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1671538562031104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5453739661824 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457314645776896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1666083450806272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1671537190468096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚450286487359488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14878456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1671537190468096 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚450286855450624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1665363552309248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1666083476979712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99656559 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1666083476979712 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1671816116233728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14880999 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5456010094080 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5454664697344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1666361451536384 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31351030 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7028938740224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50217788 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99673243 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5454664697344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1671088109894292 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16470296 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7028975366656 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450364341858304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5445331206434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1671086739865890 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1665641408659456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49455455 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5456010094080 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚457392912507392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99683236 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50217788 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7843399197983663e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1671817487803904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1666361477709824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50227775 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49455455 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99683236 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31351295 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚450363973767168 _runtime‚10080.383666679 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49455461 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50227775 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1671817487803904 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1666361477709824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49455461 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14880994 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175ÓÌ?•‰N P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1666361451536384 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚t)*øÐ0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1671816116233728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14880999 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5456010094080 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5454664697344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1666361451536384 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31351030 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7028938740224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50217788 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99673243 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5454664697344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1671088109894292 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16470296 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7028975366656 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450364341858304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5445331206434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1671086739865890 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1665641408659456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49455455 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5456010094080 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚457392912507392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99683236 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50217788 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7843399197983663e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1671817487803904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1666361477709824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50227775 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49455455 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99683236 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31351295 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚450363973767168 _runtime‚10080.383666679 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49455461 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50227775 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1671817487803904 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1666361477709824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49455461 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14880994 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚5799 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51Ü<œ œ4–/ _wandbruntime‚10080  _runtime‚10080RH§ä—‰ 1loss/SRqqt2/detection_loss‚0.6502577423174319 trainer/global_step‚5799 2loss/FBt2/assignment_loss‚0.019953697696109135 /loss/FRt1/detection_loss‚0.5095220796322665 _runtime‚10080.445111789 3loss/SRqqt1/assignment_loss‚0.48932582652873075 4loss/FBt1/assignment_loss‚1.1956071988349296e-05 %loss/total_loss‚8.014301791277951 0loss/FRt2/assignment_loss‚0.8423992285339666 2loss/SRqqt2/assignment_loss‚0.5353430875448391 /loss/FRt2/detection_loss‚1.5971352853935115 0loss/FBt1/detection_loss‚0.48512740399237964 /loss/FBt2/detection_loss‚0.5945631104864573 epoch‚14 % _timestamp‚1.7843399198599355e+09 _step‚477 0loss/FRt1/assignment_loss‚1.5649293421481423 1loss/SRqqt1/detection_loss‚0.7257330309321282’ÈJ4˜Ž 0loss/FBt1/detection_loss‚0.48512740399237964 /loss/FBt2/detection_loss‚0.5945631104864573 epoch‚14 % _timestamp‚1.7843399198599355e+09 _step‚477 0loss/FRt1/assignment_loss‚1.5649293421481423 1loss/SRqqt1/detection_loss‚0.7257330309321282 1loss/SRqqt2/detection_loss‚0.6502577423174319 trainer/global_step‚5799 2loss/FBt2/assignment_loss‚0.019953697696109135 /loss/FRt1/detection_loss‚0.5095220796322665 _runtime‚10080.445111789 3loss/SRqqt1/assignment_loss‚0.48932582652873075 4loss/FBt1/assignment_loss‚1.1956071988349296e-05 %loss/total_loss‚8.014301791277951 0loss/FRt2/assignment_loss‚0.8423992285339666 2loss/SRqqt2/assignment_loss‚0.5353430875448391 /loss/FRt2/detection_loss‚1.5971352853935115Ý^ò9ÓÖ™‚(:Ë Þ»ëÒßÌ disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚28300.75 network.recv‚ 2.526249e+06' disk.nvme0n1.out‚159380.50830078125 disk.rbd0.out‚0 disk.dm-0.in‚28300.75 cpu‚25.58260180710018) disk./.usagePercent‚40.23442722553698$ disk./.usageGB‚599.5220413208008 disk.sda3.out‚ 452.96875 disk.sda2.in‚0) proc.memory.availableMB‚ 1946.09765625* proc.memory.percent‚18.006110191345215 proc.cpu.threads‚17 network.sent‚ 1.1198664e+07 disk.nvme0n1.in‚28300.75 disk.sda1.in‚0 disk.rbd1.out‚ 27.0302734375 disk.drbd2165.out‚0 disk.sda3.in‚1.96875 disk.rbd0.in‚0 disk.sda.out‚ 452.96875 disk.dm-1.in‚1.96875 disk.dm-1.out‚ 452.96875) disk.nvme0c0n1.out‚159380.50830078125# proc.memory.rssMB‚ 2950.12109375 disk.sda2.out‚0$ memory_percent‚88.12196254730225 disk.sda.in‚1.96875$ disk.dm-0.out‚159380.50830078125v½”š‚(:‰ Þ»ëÒÝõáË gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.8678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n$pM›‚(:’ í»ëÒñÍÉÉ gpu.0.gpu‚23.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚332.025' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.40499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AA™góœ‚(:è í»ëÒøÖùË disk.sda2.in‚0 disk.sda1.in‚0 disk.sda3.in‚1.96875$ memory_percent‚88.12048435211182 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚ 2.531011e+06 disk.sda.in‚1.96875 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚28301.71484375 disk.dm-0.in‚28301.71484375$ disk.dm-0.out‚159388.76513671875 disk.sda1.out‚0 disk.dm-1.in‚1.96875 disk.drbd2165.out‚0 disk.dm-1.out‚ 453.19140625 disk.sda3.out‚ 453.19140625) proc.memory.availableMB‚ 1946.33984375$ disk./.usageGB‚599.5222587585449) disk.nvme0c0n1.out‚159388.76513671875" disk.nvme0n1.in‚28301.71484375' disk.nvme0n1.out‚159388.76513671875* proc.memory.percent‚18.006110191345215 network.sent‚ 1.1253033e+07 disk.rbd1.out‚ 27.0302734375 disk.sda.out‚ 453.19140625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2950.12109375 cpu‚25.582949059101534* disk./.usagePercent‚40.234441817966434©Üžž‚(:“ ü»ëÒ±êÇÇ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.855' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.17100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J«ïž‚(:ä ü»ëÒµùÅÉ disk.sda.in‚1.96875 disk.dm-0.in‚28412.71484375) disk./.usagePercent‚40.23445001020753 disk.sda3.out‚ 453.42578125 disk.rbd1.in‚0 disk.sda.out‚ 453.42578125 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚159399.3486328125 network.recv‚ 2.534134e+06 disk.sda3.in‚1.96875" disk.nvme0n1.in‚28412.71484375$ disk./.usageGB‚599.5223808288574 disk.dm-1.out‚ 453.42578125 disk.rbd1.out‚ 27.0927734375$ disk.nvme0c0n1.in‚28412.71484375# disk.dm-0.out‚159399.3486328125 cpu‚25.582839097098308 network.sent‚ 1.1255955e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1946.08984375* proc.memory.percent‚18.006110191345215 disk.dm-1.in‚1.96875$ memory_percent‚88.12201023101807# proc.memory.rssMB‚ 2950.12109375 proc.cpu.threads‚17 disk.drbd2165.in‚0& disk.nvme0n1.out‚159399.3486328125Ör Š“Ÿ‚(:ˆ ‹¼ëÒðûŽÉ gpu.0.gpu‚48.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚307.985' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚61.597 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>*é ‚(:Þ ‹¼ëÒâ„‹Ë disk.sda2.out‚0 disk.sda.out‚ 453.7890625 proc.cpu.threads‚17 network.sent‚ 1.1260992e+07 disk.sda.in‚1.96875' disk.nvme0c0n1.out‚159404.009765625 disk.dm-0.in‚28412.71484375 disk.sda3.in‚1.96875) proc.memory.availableMB‚ 1945.84765625 disk.drbd2165.out‚0 disk.sda1.in‚0* proc.memory.percent‚18.006110191345215$ disk.nvme0c0n1.in‚28412.71484375" disk.dm-0.out‚159404.009765625" disk.nvme0n1.in‚28412.71484375# memory_percent‚88.1234884262085 disk.dm-1.out‚ 453.7890625 disk.drbd2165.in‚0 disk.sda3.out‚ 453.7890625 disk.rbd0.in‚0% disk.nvme0n1.out‚159404.009765625 disk.sda1.out‚0# proc.memory.rssMB‚ 2950.12109375 network.recv‚ 2.537335e+06* disk./.usagePercent‚40.234441817966434 disk.sda2.in‚0 disk.rbd1.out‚ 27.0927734375 cpu‚25.582585813810695 disk.dm-1.in‚1.96875 disk.rbd1.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚599.5222587585449×–Ñž¡‚(:“ š¼ëÒ•ÑŠÈ gpu.0.gpu‚87.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚294.623' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.92459999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸òy{좂(:á š¼ëÒŸèëÉ disk.sda.in‚1.96875$ disk.dm-0.out‚159414.55029296875 disk.sda1.in‚0 disk.dm-1.in‚1.96875 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 28413.6796875) disk.nvme0c0n1.out‚159414.55029296875 disk.sda3.out‚ 453.98828125 disk.rbd0.in‚0 cpu‚25.582249643386856 network.sent‚ 1.1266086e+07 disk.dm-1.out‚ 453.98828125 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 28413.6796875! disk.nvme0n1.in‚ 28413.6796875' proc.memory.availableMB‚ 1946.078125 proc.cpu.threads‚17 network.recv‚ 2.540575e+06# disk./.usageGB‚599.522274017334 disk.rbd1.out‚ 27.1552734375' disk.nvme0n1.out‚159414.55029296875 disk.sda1.out‚0 disk.sda2.out‚0# memory_percent‚88.1220817565918* disk./.usagePercent‚40.234442841996575 disk.rbd0.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2950.12109375* proc.memory.percent‚18.006110191345215 disk.sda.out‚ 453.98828125 disk.sda3.in‚1.96875FÎ4£/ _wandbruntime‚10158  _runtime‚10158*†óȤ _step‚478 & lr-AdamW/pg1‚0.0003961656799115698 & lr-AdamW/pg2‚0.0003961656799115698 trainer/global_step‚5849 % _timestamp‚1.7843399983104875e+09 _runtime‚10158.895540707y—f­Í¥Ç _runtime‚10158.895540707 _step‚478 & lr-AdamW/pg1‚0.0003961656799115698 & lr-AdamW/pg2‚0.0003961656799115698 trainer/global_step‚5849 % _timestamp‚1.7843399983104875e+09Þ0éT4¦/ _wandbruntime‚10158  _runtime‚10158ˆÌz—(§þO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7086997719552 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1685483542904320 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1684749630671666 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1684748260643376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49857555 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50635062 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1679257641218048 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚479 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16603561 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7086961085952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49857549 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1679257615790080 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1679983487311872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5501400982016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15005601 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50635062 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1679983487311872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49857555 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50645035 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚461248173554176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚454161175834624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100502590 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1685483542904320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚10158.896995593 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1679983513485312 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5490644853296 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5501400982016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100492611 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1684748260643376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49857555 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50635062 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1679257641218048 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚479 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16603561 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7086961085952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49857549 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1679257615790080 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1679983487311872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5501400982016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15005601 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?Device_!ÇQJStatsMonitor.on_train_batch_start/active.small_pool.freed‚50635062 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1679983487311872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49857555 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50645035 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚461248173554176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚454161175834624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100502590 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1685483542904320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚10158.896995593 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1679983513485312 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5490644853296 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5501400982016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100492611 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16606063 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1685763840240128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100519288 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5502326024704 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1679535522996224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚454238294151168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16606321 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31614207 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49865761 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚461326838882816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1685027065887798 _runtime‚10160.290830009 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1680261488041984 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100519288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7088176640512 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100509295 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚454238662242304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1685763840240128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49865761 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15008144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49865755 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31614470 trainer/global_step‚5849 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50643540 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5500980627968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7088140014080 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50653527 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5492912919976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100509295 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ÑȦù'«N REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49865761 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚461326838882816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1685027065887798 _runtime‚10160.290830009 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1680261488041984 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100519288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7088176640512 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100509295 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚454238662242304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1685763840240128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49865761 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15008144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49865755 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31614470 trainer/global_step‚5849 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50643540 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5500980627968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7088140014080 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50653527 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5492912919976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100509295 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1680261514215424 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1680261488041984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16606063 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1685763840240128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100519288 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5502326024704 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1679535522996224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚454238294151168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16606321 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31614207 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154àf-ÛÅ4¬/ _wandbruntime‚10160  _runtime‚10160­¾…­‚ 0loss/FRt2/assignment_loss‚0.7437962224059775 %loss/total_loss‚8.318244384228839 % _timestamp‚1.7843399997893302e+09 2loss/SRqqt2/assignment_loss‚0.5559562362342946 1loss/FBt1/assignment_loss‚0.02332281346793779 /loss/FRt1/detection_loss‚0.5148353908401136 1loss/SRqqt1/detection_loss‚0.7874702798704597 .loss/FBt2/detection_loss‚0.697043697329481 trainer/global_step‚5849 /loss/FRt1/assignment_loss‚1.650485012613956 1loss/SRqqt1/assignment_loss‚0.478858036791769 /loss/FRt2/detection_loss‚1.6539980109746442 1loss/SRqqt2/detection_loss‚0.6666248845264178 0loss/FBt1/detection_loss‚0.48684099501082934 _runtime‚10160.374103385 2loss/FBt2/assignment_loss‚0.059012804162957934 epoch‚14 _step‚481­Mé ®‡ 0loss/FRt2/assignment_loss‚0.7437962224059775 %loss/total_loss‚8.318244384228839 % _timestamp‚1.7843399997893302e+09 2loss/SRqqt2/assignment_loss‚0.5559562362342946 1loss/FBt1/assignment_loss‚0.02332281346793779 /loss/FRt1/detection_loss‚0.5148353908401136 1loss/SRqqt1/detection_loss‚0.7874702798704597 .loss/FBt2/detection_loss‚0.697043697329481 trainer/global_step‚5849 /loss/FRt1/assignment_loss‚1.650485012613956 1loss/SRqqt1/assignment_loss‚0.478858036791769 /loss/FRt2/detection_loss‚1.6539980109746442 1loss/SRqqt2/detection_loss‚0.6666248845264178 0loss/FBt1/detection_loss‚0.48684099501082934 _runtime‚10160.374103385 2loss/FBt2/assignment_loss‚0.059012804162957934 epoch‚14 _step‚481áU¨¶ã¯‚(:… ©¼ëÒÏ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚293.8' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.76 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;l¦î°‚(:ã ©¼ëÒ†ó•Ê disk.dm-0.in‚28414.51171875& disk.nvme0n1.out‚159422.1162109375 disk.sda2.in‚0 disk.dm-1.in‚1.96875 disk.drbd2165.out‚0$ memory_percent‚88.12875747680664 disk.drbd2165.in‚0$ disk./.usageGB‚599.5226135253906 disk.rbd0.in‚0 disk.sda.out‚ 454.25390625 network.sent‚ 1.1283956e+07" disk.nvme0n1.in‚28414.51171875 disk.sda3.in‚1.96875$ disk.nvme0c0n1.in‚28414.51171875# proc.memory.rssMB‚ 2950.12109375 cpu‚25.581955310239497 network.recv‚ 2.544291e+06# disk.dm-0.out‚159422.1162109375 disk.sda1.out‚0 disk.sda3.out‚ 454.25390625 disk.dm-1.out‚ 454.25390625 disk.rbd1.out‚ 27.1552734375( disk.nvme0c0n1.out‚159422.1162109375* proc.memory.percent‚18.006110191345215* disk./.usagePercent‚40.234465626667124 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 1944.984375 disk.rbd1.in‚0 disk.sda.in‚1.96875 proc.cpu.threads‚17£+¢èŸ±‚(:” ¸¼ëÒçǧÇ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚291.239' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.24779999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0̘5t鲂(:Þ ¸¼ëÒ¢øÉ cpu‚25.581664646409532$ memory_percent‚88.12918663024902$ disk./.usageGB‚599.5253067016602 disk.sda.out‚ 454.7890625$ disk.dm-0.out‚159431.12451171875 disk.sda1.in‚0 disk.sda2.in‚0 network.recv‚ 2.548277e+06 disk.sda.in‚1.96875 disk.rbd0.in‚0* proc.memory.percent‚18.006110191345215 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 28895.046875 proc.cpu.threads‚17' disk.nvme0n1.out‚159431.12451171875 disk.sda3.out‚ 454.7890625( proc.memory.availableMB‚ 1944.9140625# proc.memory.rssMB‚ 2950.12109375 network.sent‚ 1.1335759e+07 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚159431.12451171875 disk.dm-1.in‚1.96875 disk.dm-0.in‚ 28895.046875 disk.sda1.out‚0 disk.sda3.in‚1.96875 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 28895.046875 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 27.2177734375* disk./.usagePercent‚40.234646367986294 disk.dm-1.out‚ 454.7890625[S ð³‚(:å ǼëÒȤŒÉ) disk.nvme0c0n1.out‚159439.41064453125 network.sent‚ 1.1337977e+07 disk.sda3.out‚ 455.29296875 disk.rbd0.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2950.12109375$ disk.dm-0.out‚159439.41064453125 disk.drbd2165.in‚0 disk.drbd2165.out‚0' disk.nvme0n1.out‚159439.41064453125 disk.rbd1.in‚0 cpu‚25.581378161931706$ disk./.usageGB‚599.5254287719727$ disk.nvme0c0n1.in‚28896.04296875 disk.sda1.out‚0 disk.dm-1.in‚1.96875 proc.cpu.threads‚17$ memory_percent‚88.12918663024902) disk./.usagePercent‚40.23465456022739 disk.sda1.in‚0 disk.dm-1.out‚ 455.29296875 disk.rbd1.out‚ 27.2177734375 network.recv‚ 2.55146e+06 disk.dm-0.in‚28896.04296875" disk.nvme0n1.in‚28896.04296875 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda.in‚1.96875 disk.sda.out‚ 455.29296875* proc.memory.percent‚18.006110191345215 disk.sda3.in‚1.96875( proc.memory.availableMB‚ 1944.9140625’7l ”´‚(:‰ ǼëÒ£æ”É gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚270.414' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íi¼Ÿµ‚(:” Ö¼ëÒÉè—Ç gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚335.114' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.02279999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06òsÈݶ‚(:Ò Ö¼ëÒÿ´¯É disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 455.484375 disk.sda.in‚1.96875 disk.dm-1.in‚1.96875 disk.drbd2165.out‚0 network.sent‚ 1.134306e+07 disk.dm-0.in‚ 28896.875) proc.memory.availableMB‚ 1944.91015625 cpu‚25.581111001762096* disk./.usagePercent‚40.234660448400675 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 28896.875) disk.nvme0c0n1.out‚159452.79541015625$ disk.dm-0.out‚159452.79541015625 disk.sda1.in‚0# proc.memory.rssMB‚ 2950.12109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 455.484375 disk.sda3.in‚1.96875$ memory_percent‚88.12921047210693* proc.memory.percent‚18.006110191345215 proc.cpu.threads‚17 disk.sda.out‚ 455.484375 disk.rbd0.in‚0 network.recv‚ 2.554765e+06$ disk./.usageGB‚599.5255165100098 disk.rbd1.out‚ 27.3427734375 disk.nvme0n1.in‚ 28896.875' disk.nvme0n1.out‚159452.79541015625S(}¢·‚(:” å¼ëÒ„ç¾È gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpjþò«ýu.0.temp‚44.0 gpu.0.powerWatts‚263.406' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.681200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºz߸‚(:Ô å¼ëÒÐÅ’É disk.dm-0.in‚ 28896.875$ disk.dm-0.out‚159457.08056640625' disk.nvme0n1.out‚159457.08056640625 disk.sda1.out‚0 disk.sda.out‚ 455.7890625) disk.nvme0c0n1.out‚159457.08056640625 network.recv‚ 2.558096e+06 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd2165.in‚0* proc.memory.percent‚18.006110191345215 network.sent‚ 1.1348039e+07 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 455.7890625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 28896.875 disk.dm-1.in‚1.96875 disk.drbd2165.out‚0 disk.sda3.in‚1.96875 disk.rbd1.out‚ 27.3427734375 disk.sda.in‚1.96875# proc.memory.rssMB‚ 2950.12109375 disk.nvme0c0n1.in‚ 28896.875$ memory_percent‚88.13021183013916 cpu‚25.580822907951674# disk./.usageGB‚599.525447845459 disk.sda3.out‚ 455.7890625 disk.sda2.out‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 1944.74609375) disk./.usagePercent‚40.23465584026506e|í4¹/ _wandbruntime‚10238  _runtime‚10238–ȵ漀 _step‚482 ' lr-AdamW/pg1‚0.00039467879258226455 ' lr-AdamW/pg2‚0.00039467879258226455 trainer/global_step‚5899 % _timestamp‚1.7843400779317975e+09 _runtime‚10238.516809302´([rÏ»É ' lr-AdamW/pg2‚0.00039467879258226455 trainer/global_step‚5899 % _timestamp‚1.7843400779317975e+09 _runtime‚10238.516809302 _step‚482 ' lr-AdamW/pg1‚0.00039467879258226455â8)[¹4¼/ _wandbruntime‚10238  _runtime‚102382ë`ù(½ýO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5536881966404 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1693883523817472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚465181556029952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7146059811328 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1693151730126848 _step‚483 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101338642 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1693883549990912 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1699431266903552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1693883549990912 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5547716912640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5546371523072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚465181151305216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51070787 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51070787 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51060814 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7146023177728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15132751 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1698689956693574 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚5899 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1699431266903552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1699429895340544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101328663 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5547716912640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5546371523072 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1693883523817472 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51060814 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1698688586665284 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16740414 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5538226566726 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16740686 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚458035128127488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _runtime‚10238.518025909 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50267849 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15132756 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458035496218624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50267855 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31873165 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 $ _timestamp‚1.784340077932828e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5547716912640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5546371523072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚465181151305216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51070787 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51070787 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51060814 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7146023177728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15132751 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1698689956693574 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚5899 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1699431266903552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1699429895340544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101328663 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5547716912640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5546371523072 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1693883523817472 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51060814 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1698688586665284 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16740414 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5538226566726 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16740686 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚458035128127488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _runtime‚10238.518025909 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50267849 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15132756 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458035496218624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ãĈ´Ô4¿/ _wandbruntime‚10239  _runtime‚10239=g~ë'ÀˆN ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1699708821106176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51079279 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50276055 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458112982626304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15135299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16743175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7843400793297203e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50276061 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50276055 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1694161524547584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5547296558592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16743434 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7147238403584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5539150033084 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚10239.914944804 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1699710192676352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101355340 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7147201777152 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1698967391909706 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1693429586477056 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚484 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101355340 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1694161550721024 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31878733 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚465259816312320 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51079279 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31878469 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101345347 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5547296558592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51069292 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1698968761938108 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5548641955328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚458112614535168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1693429611905024 trainer/global_step‚5899 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1699710192676352ñÀ”‰ÁN M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1699708821106176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51079279 M=DeviceStatsMonitor.on_train_batch_end/reserved_<ãR·Š%bytes.all.peak‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50276055 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458112982626304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15135299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16743175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7843400793297203e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50276061 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50276055 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1694161524547584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5547296558592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16743434 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7147238403584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5539150033084 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚10239.914944804 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1699710192676352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101355340 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7147201777152 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1698967391909706 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1693429586477056 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚484 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101355340 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1694161550721024 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31878733 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚465259816312320 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51079279 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31878469 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101345347 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5547296558592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51069292 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1698968761938108 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5548641955328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚458112614535168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1693429611905024 trainer/global_step‚5899 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1699710192676352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 QDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1707783586496512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚469115025815040 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1713376247776768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚469115428442624 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1707045793607680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5583119079512 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51496539 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15259901 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 trainer/global_step‚5949 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51486566 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50678149 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1713376247776768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5594032843264 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102174694 _step‚487 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51486566 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15259906 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚461909816602624 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _runtime‚10317.960107445 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1713377619339776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16877680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1713377619339776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1707783560323072 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5592687453696 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102164715 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5584463679834 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1712630282715482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51496539 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚461909448511488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7205577303552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7205611840000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1707783586496512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚469115025815040 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1713376247776768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚469115428442624 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1707045793607680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5583119079512 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51496539 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15259901 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 trainer/global_step‚5949 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51486566 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50678149 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1713376247776768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5594032843264 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102174694 _step‚487 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51486566 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15259906 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚461909816602624 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _runtime‚10317.960107445 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1713377619339776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16877680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1713377619339776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1707783560323072 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5592687453696 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102164715 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5584463679834 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1712630282715482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51496539 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚461909448511488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7205577303552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7205611840000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5593612489216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5593612489216 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1707323700813824 $ _timestamp‚1.784340158768651e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50686361 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1708061561053184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7206798445056 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1712909087960016 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _runtime‚10319.354002891 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5949 _step‚488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32142570 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1713656545112576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1708061587226624 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1707323675385856 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1708061587226624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1713656545112576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚469193698834944 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102181399 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51495044 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16880126 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102191392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416ÑL'׌N F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1707323700813824 $ _timestamp‚1.784340158768651e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50686361 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1708061561053184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7206798445056 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1712909087960016 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _runtime‚10319.354002891 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5949 _step‚488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32142570 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1713656545112576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1708061587226624 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1707323675385856 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1708061587226624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1713656545112576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚469193698834944 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102181399 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51495044 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16880126 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102191392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50686355 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚469194101455360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5593612489216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5593612489216 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5èwp54Ø/ _wandbruntime‚10319  _runtime‚10319mz:¸ Ù„ /loss/FRt1/detection_loss‚0.4758232868187221 0loss/SRqqt2/detection_loss‚0.677585038540287 /loss/FBt2/detection_loss‚0.6586407693102956 0loss/FRt2/assignment_loss‚0.8232947516797194 2loss/SRqqt1/assignment_loss‚0.4175648051480462 1loss/SRqqt1/detection_loss‚0.6982197301215407 $loss/total_loss‚8.08231127965209 0loss/FRt1/assignment_loss‚1.5733075113038393 2loss/FBt1/assignment_loss‚0.001285148094717785 3loss/FBt2/assignment_loss‚0.0006190799918295298 /loss/FRt2/detection_loss‚1.6612102074933177 trainer/global_step‚5949 % _timestamp‚1.7843401588521686e+09 2loss/SRqqt2/assignment_loss‚0.5440925592661511 /loss/FBt1/detection_loss‚0.5506683918836249 epoch‚14 _runtime‚10319.436970332 _step‚489ÿª2Ú‰ /loss/FRt1/detection_loss‚0.4758232868187221 0loss/SRqqt2/detection_loss‚0.677585038540287 /loss/FBt2/detection_loss‚0.6586407693102956 0loss/FRt2/assignment_loss‚0.8232947516797194 2loss/SRqqt1/assignment_loss‚0.4175648051480462 1loss/SRqqt1/detection_loss‚0.6982197301215407 $loss/total_loss‚8.08231127965209 0loss/FRt1/assignment_loss‚1.5733075113038393 2loss/FBt1/assignment_loss‚0.001285148094717785 3loss/FBt2/assignment_loss‚0.0006190799918295298 /loss/FRt2/detection_loss‚1.6612102074933177 trainer/global_step‚5949 % _timestamp‚1.7843401588521686e+09 2loss/SRqqt2/assignment_loss‚0.5440925592661511 /loss/FBt1/detection_loss‚0.5506683918836249 epoch‚14 _runtime‚10319.436970332 _step‚489éV˜¦ïÛ‚(:ä ¿½ëÒ°ä•Ì disk.dm-1.out‚ 457.82421875 disk.sda3.in‚1.96875 proc.cpu.threads‚17) proc.memory.availableMB‚ 1945.92578125 network.recv‚ 2.580496e+06) disk./.usagePercent‚40.23464585597122 disk.sda.out‚ 457.82421875 disk.drbd2165.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0" disk.nvme0n1.in‚29106.66796875' disk.nvme0n1.out‚159501.46923828125 disk.sda1.out‚0) disk.nvme0c0n1.out‚159501.46923828125$ disk.dm-0.out‚159501.46923828125# proc.memory.rssMB‚ 2950.12109375 cpu‚25.57911332909517 disk.sda.in‚1.96875$ disk.nvme0c0n1.in‚29106.66796875$ disk./.usageGB‚599.5252990722656 disk.rbd0.in‚0 disk.rbd1.out‚ 27.5302734375 disk.drbd2165.out‚0 disk.dm-0.in‚29106.66796875 disk.sda3.out‚ 457.82421875* proc.memory.percent‚18.006110191345215# memory_percent‚88.1230115890503 network.sent‚ 1.143594e+07 disk.sda2.out‚0 disk.dm-1.in‚1.96875 disk.rbd0.out‚0 disk.sda2.in‚0P$ žÜ‚(:’ ¿½ëÒäÖ¤Ì gpu.0.gpu‚19.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.618' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.72359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cé6¹”Ý‚(:‰ νëÒ±ÐüÈ gpu.0.gpu‚97.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚271.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.2956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì-ÔÝÞ‚(:Ò νëÒèñ×É disk.sda2.in‚0 disk.rbd1.out‚ 27.5927734375 disk.sda1.out‚0$ disk./.usageGB‚599.5254516601562 disk.sda2.out‚0! disk.nvme0n1.in‚ 29107.6328125' disk.nvme0n1.out‚159510.86865234375 disk.sda1.in‚0$ memory_percent‚88.12601566314697* disk./.usagePercent‚40.234656096272595 disk.sda.in‚1.96875 network.sent‚ 1.1499132e+07 disk.dm-1.out‚458.25$ disk.dm-0.out‚159510.86865234375 disk.sda3.in‚1.96875) proc.memory.availableMB‚ 1945.43359375 disk.dm-0.in‚ 29107.6328125 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2950.12109375 cpu‚25.578818558499417 proc.cpu.threads‚17 disk.dm-1.in‚1.96875 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 29107.6328125 disk.sda.out‚458.25 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚159510.86865234375* proc.memory.percent‚18.006110191345215 network.recv‚ 2.58513e+06 disk.drbd2165.in‚0 disk.sda3.out‚458.25°ß#ý’ß‚(:‡ ݽëÒä×àÈ gpu.0.gpu‚64.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚329.76' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xÄŠâà‚(:× ݽëÒúÏÞÊ network.recv‚ 2.588599e+06 disk.dm-1.in‚1.96875 disk.rbd1.in‚0 disk.sda2.in‚0) disk./.usagePercent‚40.23461615909725$ disk./.usageGB‚599.5248565673828 disk.drbd2165.out‚0 disk.rbd0.out‚0$ memory_percent‚88.12453746795654 disk.sda.out‚ 458.66015625 disk.dm-0.out‚159514.1328125! disk.nvme0n1.in‚ 29107.6328125 disk.sda1.in‚0 disk.rbd1.out‚ 27.5927734375 disk.drbd2165.in‚0 disk.sda3.out‚ 458.66015625 disk.sda2.out‚0# proc.memory.rssMB‚ 2950.12109375 proc.cpu.threads‚17 disk.dm-1.out‚ 458.66015625 disk.sda.in‚1.96875% disk.nvme0c0n1.out‚159514.1328125 disk.dm-0.in‚ 29107.6328125 disk.rbd0.in‚0 cpu‚25.578554629760777# disk.nvme0n1.out‚159514.1328125* proc.memory.percent‚18.006110191345215# disk.nvme0c0n1.in‚ 29107.6328125 disk.sda1.out‚0 disk.sda3.in‚1.96875 network.sent‚ 1.150609e+07) proc.memory.availableMB‚ 1945.67578125Êý`†˜á‚(:‰ ì½ëÒŽÉòÇ gpu.0.gpu‚88.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 4961658u3œˆü4704 gpu.0.temp‚44.0 gpu.0.powerWatts‚273.389' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èõؤãâ‚(:Ø ì½ëÒ°¥èÉ proc.cpu.threads‚17 disk.sda3.out‚458.875$ memory_percent‚88.12572956085205 disk.dm-1.out‚458.875 disk.rbd1.in‚0 disk.dm-0.in‚29108.46484375" disk.nvme0n1.in‚29108.46484375 disk.sda1.out‚0 disk.sda3.in‚1.96875) disk./.usagePercent‚40.23495229698971$ disk./.usageGB‚599.5298652648926) disk.nvme0c0n1.out‚159526.07568359375# proc.memory.rssMB‚ 2950.12109375 disk.sda.in‚1.96875 disk.drbd2165.in‚0' disk.nvme0n1.out‚159526.07568359375 disk.sda2.in‚0) proc.memory.availableMB‚ 1945.48046875 network.recv‚ 2.592026e+06 disk.sda.out‚458.875 disk.drbd2165.out‚0$ disk.dm-0.out‚159526.07568359375 disk.sda1.in‚0 cpu‚25.578277511126043 disk.rbd0.in‚0 disk.dm-1.in‚1.96875$ disk.nvme0c0n1.in‚29108.46484375 network.sent‚ 1.1509113e+07 disk.sda2.out‚0* proc.memory.percent‚18.006110191345215 disk.rbd1.out‚ 27.6552734375 disk.rbd0.out‚0×qÔžã‚(:“ û½ëÒ؇»É gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚290.855' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.17100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìrræä‚(:Û û½ëÒ¢¦ÑË disk.dm-0.in‚29212.65234375$ disk.dm-0.out‚159534.11767578125 disk.sda3.in‚1.96875) disk./.usagePercent‚40.23499300218766$ disk./.usageGB‚599.5304718017578 disk.rbd1.in‚0 disk.drbd2165.out‚0" disk.nvme0n1.in‚29212.65234375 disk.sda1.in‚0# proc.memory.rssMB‚ 2950.12109375 disk.rbd0.in‚0 disk.sda2.out‚0' disk.nvme0n1.out‚159534.11767578125 disk.sda3.out‚459.1875 disk.rbd0.out‚0 disk.sda.in‚1.96875$ disk.nvme0c0n1.in‚29212.65234375$ memory_percent‚88.12439441680908 cpu‚25.577963141023915 disk.sda1.out‚0 disk.dm-1.out‚459.1875* proc.memory.percent‚18.006110191345215 network.sent‚ 1.1516188e+07 disk.rbd1.out‚ 27.6552734375 proc.cpu.threads‚17 disk.sda.out‚459.1875 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚159534.11767578125 disk.sda2.in‚0 disk.dm-1.in‚1.96875) proc.memory.availableMB‚ 1945.69921875 network.recv‚ 2.595711e+06\Š`ƒ”å‚(:‰ Š¾ëÒªíÚÇ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚271.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.2956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xYñæ‚(:æ Š¾ëÒ¹òÐÉ" disk.nvme0n1.in‚29212.65234375 disk.rbd0.out‚0# proc.memory.rssMB‚ 2950.12109375' disk.nvme0n1.out‚159538.59033203125 disk.drbd2165.in‚0( proc.memory.availableMB‚ 1945.4453125$ disk./.usageGB‚599.5305519104004 disk.sda1.in‚0 disk.sda.in‚1.96875$ disk.nvme0c0n1.in‚29212.65234375 disk.dm-0.in‚29212.65234375 proc.cpu.threads‚17 disk.sda1.out‚0$ memory_percent‚88.12594413757324) disk./.usagePercent‚40.23499837834588 disk.dm-1.out‚ 459.48046875 disk.rbd1.in‚0 disk.sda.out‚ 459.48046875 disk.drbd2165.out‚0* proc.memory.percent‚18.006110191345215 disk.sda3.out‚ 459.48046875 disk.sda2.in‚0 disk.dm-1.in‚1.96875 disk.rbd1.out‚ 27.7177734375$ disk.dm-0.out‚159538.59033203125 cpu‚25.577693212405567 network.recv‚ 2.599177e+06 disk.rbd0.in‚0) disk.nvme0c0n1.out‚159538.59033203125 network.sent‚ 1.1519253e+07 disk.sda3.in‚1.96875 disk.sda2.out‚0P«è²4ç/ _wandbruntime‚10397  _runtime‚10397|q¨…Éèà _step‚490 ' lr-AdamW/pg1‚0.00039167554815964546 ' lr-AdamW/pg2‚0.00039167554815964546 trainer/global_step‚5999 $ _timestamp‚1.784340236998138e+09 _runtime‚10397.583169947)GÅÎéÈ trainer/global_step‚5999 $ _timestamp‚1.784340236998138e+09 _runtime‚10397.583169947 _step‚490 ' lr-AdamW/pg1‚0.00039167554815964546 ' lr-AdamW/pg2‚0.00039167554815964546êÐ4ê/ _wandbruntime‚10397  _runtime‚10397ž*JÈ(ëýO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103000767 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1721683596828672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1721683623002112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚465784136986624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51912318 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17014264 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51922291 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1727322600212992 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5639003384320 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1721683596828672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32401078 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17014027 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1727323971776000 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1721683623002112 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1727323971776000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103010746 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15387051 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5629356192620 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ðú¶í(ì‚P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15387051 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5629356192620 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103000767 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1721683596828672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1721683623002112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚465784136986624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51912318 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17014264 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51922291 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1727322600212992 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5639003384320 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1721683596828672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32401078 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17014027 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1727323971776000 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1721683623002112 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1727323971776000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103010746 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32406333 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5631624259300 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51096661 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1721961623732224 _runtime‚10398.978533859 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51920796 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1721217789722624 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51096655 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51930783 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7266831944192 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32406570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚473128487770624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 % _timestamp‚1.7843402383933299e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103027444 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103017451 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51096655 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15389599 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1727601525978624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15389594 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7266864376320 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1726849413981924 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚465861255303168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103027444 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1727602897548800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1726848043953522 trainer/global_step‚5999 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5641273816576 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1727602897548800 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820gvüŸ'ïN N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32406570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚473128487770624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 % _timestamp‚1.7843402383933299e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103027444 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103017451 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51096655 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15389599 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1727601525978624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15389594 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7266864376320 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1726849413981924 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚465861255303168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103027444 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1727602897548800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1726848043953522 trainer/global_step‚5999 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5641273816576 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1727602897548800 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32406333 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5631624259300 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51096661 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1721961623732224 _runtime‚10398.978533859 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51920796 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1721217789722624 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51096655 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51930783 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7266831944192 B&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08973036706447601 *jet/accuracy_1_of_1‚0.3290625512599945 *jet/accuracy_1_of_2‚0.7901392579078674 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2790246605873108 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5499359369277954 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1637279987335205 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4010416567325592 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 (loss/val_total_loss‚8.12255973449741 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.74894779920578 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14213326573371887 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3107059597969055 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1517632156610489 jet/accuracy_1_of_5‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2440054565668106 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6392823457717896 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04189067333936691 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.29455986618995667 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1517632156610489 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.31976062059402466 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9934337139129639 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_1‚1 _runtime‚10425.248840655 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9878772497177124 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4513888955116272 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 _step‚494 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.14475271105766296 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5447912812232971 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07421831041574478 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4888419210910797 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9977869391441345 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.827532172203064 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6667431592941284 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5362033843994141 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3025059700012207 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4861311614513397 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5840906500816345 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6247515678405762 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.26768168807029724 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.682968020439148 particle/accuracy_3_of_4‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33724576234817505 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 jet/accuracy_4_of_5‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.27815380692481995 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8685089349746704 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6393243074417114 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.3459146022796631 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6164724826812744 jet/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46027329564094543 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6318694949150085 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.35146304965019226 *jet/accuracy_1_of_4‚0.9647459983825684 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3120482861995697 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7962962985038757 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458134651184 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.32192131876945496 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17083333432674408 (particle/accuracy‚0.8503844141960144 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3289180099964142 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3087359666824341 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 /particle/accuracy_4_of_4‚0.9545367360115051 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6735618710517883 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6399617195129395 6validation_average_jet_accuracy‚0.3704167902469635 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6336402893066406 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +particle/sensitivity‚0.6393784284591675 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3636761009693146 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6358950734138489 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5797503590583801 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6412017941474915 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7421464920043945 *jet/accuracy_2_of_4‚0.7906146049499512 particle/accuracy_3_of_3‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6162926554679871 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.011098379269242287 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7258739471435547 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976841807365417 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04658323526382446 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7962962985038757 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5882514715194702 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9955822229385376 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7019171118736267 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.30056461691856384 +jet/accuracy_3_of_4‚0.08414210379123688 trainer/global_step‚5999 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6357235312461853 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_2_of_2‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.28898701071739197 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5795367360115051 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23531123995780945 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5753699541091919 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6630695462226868 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.054002661257982254 +jet/accuracy_2_of_2‚0.14218580722808838 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930034875869751 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 jet/accuracy_5_of_5‚0 epoch‚14 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10401935875415802 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5670241117477417 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998410165309906 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5927615761756897 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.631691575050354 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9957568049430847 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5408193469047546 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8206161260604858 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3120482861995697 particle/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9537036418914795 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +particle/specificity‚0.9080938696861267 particle/accuracy_5_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6476569175720215 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8080499768257141 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.32120856642723083 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 particle/accuracy_2_of_3‚1 % _timestamp‚1.7843402646631134e+09 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961844086647034 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7942177057266235 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5899722576141357 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969257116317749 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5756692290306091 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5185028314590454 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/1FRt1SRqqt0FBt/event